feat(cli): show the prompt-cache hit rate in the TUI during a run
Problem
#8964 landed the counters and a /context readout, but not a hit rate. Three gaps remain between what the TUI shows today and the epic's acceptance criterion — "the live hit rate is visible to the user during a run."
1. There is no session aggregate
The TUI holds exactly two cache numbers, and both are per-turn:
src-tauri/src/core/cli/tui.rs:2100-2101
turn_cached_tokens: u64,
turn_cache_write_tokens: u64,They are zeroed on every turn — tui.rs:2637-2638, tui.rs:2667-2668, tui.rs:5389-5390 — and the rendered line says so:
src-tauri/src/core/cli/tui.rs:6112
"Prompt cache (last request): {read} read ({read_pct:.0}% of prompt), {} written"One request is a sample, not a rate. The number the epic is about — what fraction of this session's prompt tokens were served from cache — is never computed. A session that hits 95% for twenty turns and then thrashes shows the same last-request line as one that thrashed throughout.
2. It is hidden in exactly the case worth seeing
src-tauri/src/core/cli/tui.rs:4800-4801
let cache_reported =
reported && (snapshot.turn_cached_tokens > 0 || snapshot.turn_cache_write_tokens > 0);cache_summary_line returns None when cache_reported is false (tui.rs:6098). So a provider that does report cache usage and honestly reports zero tokens read renders identically to a provider that reports nothing at all: the line vanishes.
provider reports 90K read → "Prompt cache (last request): 90K read (75% of prompt)"
provider reports 0 read → (nothing) ← the alarm condition
provider reports nothing → (nothing) ← indistinguishableA 0% hit rate is the single most expensive state the agent can be in, and it is the one state the UI refuses to draw. On a write-billing provider it is worse than silence: a prefix written every turn and never read is a 25% surcharge on the whole prompt, and the user sees no line at all.
context_view_hides_prompt_cache_when_not_reported (tui.rs:29988) pins the not-reported case. Nothing pins the zero-hit case, because today they are the same code path.
3. It is on-demand only
The readout exists solely inside /context. Nothing appears during a run. A user watching a long session burn money has to interrupt and type a slash command to find out — and per (2), may get a blank.
Proposed change
Accumulate per session, render a rate, and separate "zero" from "unknown."
- Add session-cumulative counters alongside the per-turn ones — cached, written, and total prompt tokens — that are not reset by
reset_turn_usage. The rate issession_cached / session_prompt, not a mean of per-turn percentages. - Split the gate.
cache_reportedshould mean the route reports cache usage, not the route reported a non-zero number this turn. Ideally derived from #8963'sreports_prompt_cache_usage, which does not exist yet; until it does, latch "this route has reported at least once this session" rather than testing the current turn's value. - Render
0%as0%, and a non-reporting route asnot reported. - Surface the session rate during a run, not only under
/context— the status line is the obvious home.
Suggested shape for /context:
Prompt cache (session): 1.2M read (91% of prompt), 140K written
Prompt cache (last request): 90K read (75% of prompt)and on a route that reports nothing:
Prompt cache: not reported by this providerAcceptance criteria
- Session-cumulative cache read / write / prompt token counters exist and survive turn boundaries.
-
/contextshows a session hit rate as a share of session prompt tokens, alongside the existing last-request line. - A reporting route with zero cache reads renders
0%, and a test pins that it is visually distinct from the non-reporting case. - A non-reporting route renders
not reported, never a fabricated0%— and the existingcontext_view_hides_prompt_cache_when_not_reportedbehaviour is preserved or consciously replaced. - The live session hit rate is visible during a run without typing a slash command.
- Resuming a session does not report a hit rate that silently excludes the pre-resume turns, or it states that it is scoped to this process.
Related
- Closes the gap left by #8964, whose criterion "The TUI shows a per-session hit rate; a non-reporting route shows 'not reported' rather than 0%" was not met by #8967.
- Depends on #8963 for a clean
reports_prompt_cache_usagesignal; workable without it, at the cost of a latch heuristic. - The same
Some(0)/Noneconflation exists in the CLI JSON envelope —src-tauri/src/core/cli/run_report.rs:69-70aggregates withunwrap_or(0). Worth fixing in the same pass. - Prior art: DeepSeek-Reasonix puts the live hit rate in the TUI; oh-my-pi splits
cacheRead/cacheWritecost per provider. oh-my-pi#11897 is the case for visibility — two prompt injectors caused an 80.2% overspend with 98.7% of input tokens written to cache and never read, which a visible read-share would have shown immediately.
Part of #8956.
Source: janhq/jan