Pluggable cache-aware routing policies + eval harness
Author: adilhafeezCreated Jul 21, 2026Updated Jul 21, 2026
Labelsenhancementarea/routing
Follow-up from #982 review (raised by @knn-do).
The routing budget shipped in #982 is one specific cache-aware policy: stick to the warm anchor, gate paid switches by a cumulative overhead cap. The review raised that this policy is sensitive to the initial prompt and can hold a session on a sub-optimal route for an extended period, and asked for two things this issue tracks:
- Policy abstraction. Structure the session-routing decision (
session_router::route) so alternative cache-aware policies can be implemented and selected via config — e.g. decaying stickiness, quality-delta thresholds, per-route budgets — without touching the decision plumbing (warmth inference, binding persistence, telemetry stay shared). - Fast policy evaluation. A way to test candidate policies against recorded traffic and measure quality/cost tradeoffs systematically, instead of building docs/observability per policy. Building blocks already exist:
record_counterfactual(the road not taken),plano.session.overhead_pct/baseline_in_usd/switch_spend_in_usdspan attributes, andbrightstaff_session_switch_decisions_total.
Open product question from the same thread (for @salmanap): maintain multiple strategies in Plano long-term, or evaluate and keep only the best one?
Source: katanemo/plano