Improve switch-cost accuracy: cross-model tokenizer variance
Author: adilhafeezCreated Jul 21, 2026Updated Jul 21, 2026
Labelsenhancementarea/routing
Follow-up from #982 review (raised by @knn-do).
The switch-cost gate prices a switch as context_tokens x (candidate_uncached_rate - anchor_cached_rate), using a single context-token count for both sides. But tokenizers differ materially across model families (OpenAI's is notably more efficient than Anthropic's and most open models), so the same context can be 10-30% more tokens on one side of the switch than the other.
Current state after #982:
actual_context_tokens()counts with the tiktoken-based counter for the request model, falling back to chars/4.- On the full-proxy path the binding is refined with the provider's real reported prompt-token count, which fixes the anchor side over time — but the candidate side is still priced with the anchor-derived count.
- The A->B->A case is already handled (bounded route history + warm-token credit, #982).
Possible directions (needs data before committing):
- Count context per-side with each model's own tokenizer when known.
- Apply a per-family correction factor derived from observed usage.
- Measure how large the error actually is in practice from recorded usage before adding complexity.
Source: katanemo/plano