epic: prefix-cache discipline for Jan Agent

Jan Agent re-sends a few thousand identical leading tokens on every turn — system prompt, tool schemas, accepted history. Providers reuse that region only when the leading bytes are byte-identical to the previous request. Jan changes those bytes on nearly every turn, so most of that region is re-billed at full price.

This epic collects the changes that make the head of a Jan Agent request hold still.

Why the hit rate is the bill

DeepSeek published price card, peak rate, deepseek-flash:

cache hit     $0.006 / M   ▎
cache miss    $0.300 / M   ██████████████████████████████████████████████████   50×
output        $1.200 / M

A miss costs 50× a hit. So on a long agent session the hit rate — not the token count, not the model choice — sets the invoice:

100 calls · 40k prompt tokens each · 1k output tokens each

  no reuse    4.0M × $0.30  = $1.200   +  0.1M × $1.20 = $0.120   →   $1.320
  90% reuse   3.6M × $0.006 = $0.022   +  0.4M × $0.30 = $0.120
                                       +  0.1M × $1.20 = $0.120   →   $0.262

                                                          saving   −80.2%

On providers that bill cache writes (Anthropic, Bedrock), a prefix that is rewritten every turn is worse than no cache at all: a write costs 1.25× uncached input and a read 0.10×, so a written-never-read prefix is a 25% surcharge on the whole prompt.

Anatomy of a Jan Agent request today

        ┌───────────────── reusable prefix — should never change ─────────────────┐  ┌── tail ──┐
        │                                                                          │  │          │
 wire   │  "Today's date is 2026-09-16."  │  system prompt  │  memory block  │ tools │  │ history  │
        │              ▲                  │        ▲        │       ▲        │   ▲   │  │          │
        └──────────────┼──────────────────┴────────┼────────┴───────┼────────┴───┼───┘  └──────────┘
                       │                           │                │            │
              rotates at midnight,        replaced wholesale   recomputed     HashMap
              and differs per timezone      on every run      per user query  iteration
                                                                               order
              loop.rs:2022-2031           upstream.rs:312-321  loop.rs:2008-2021  upstream.rs:599-621
                   (#8957)                      (#8960)             (#8958)          (#8959)

Four independent writers above the cache line. Any one of them changing invalidates everything to its right.

What one turn looks like

sequenceDiagram
    autonumber
    participant J as Jan Agent
    participant P as Provider cache

    Note over J: turn 1 · 09:00
    J->>P: [date=09-16][system][memory(q1)][tools] + history
    P-->>J: cold miss — whole prompt billed at full price

    Note over J: turn 2 · 09:01 — only the user text changed
    J->>P: [date=09-16][system][memory(q2)][tools] + history
    P-->>J: MISS again — memory block differs at byte ~2,400

    Note over J: turn 3 · next day, or a different timezone
    J->>P: [date=09-17][system][memory(q3)][tools] + history
    P-->>J: MISS at byte 0 — nothing after it can be reused

The correct shape is one cold miss per session, and a hit on every turn after it.

Sub-issues

#8957–#8960 are the cheap, high-value ones. #8964 is the one that keeps working after the rest are done, because a provider cache is best-effort and client discipline alone cannot prove a hit.

Prior art — four other agents already paid for these lessons

Codex is the load-bearing lesson: a perfect client still loses money if it cannot see the provider's cache. That is why #8964 is not optional.

Jan is also the only one of the five that must satisfy three different cache models in one build — DeepSeek's automatic prefix units, OpenAI-compatible implicit reuse, and local llama.cpp — which is exactly why #8963 exists.

Acceptance for the epic

• A session of N turns on a DeepSeek route shows one cold miss and a hit on every subsequent turn, measured from prompt_cache_hit_tokens / prompt_cache_miss_tokens.
• Restarting the app and resuming a session does not produce a cold miss caused by client-side ordering.
• Crossing midnight mid-session does not invalidate the prefix.
• Compaction costs exactly one scheduled prefix break, not a rejected request followed by a retry.
• The live hit rate is visible to the user during a run.
• A regression test fails if any new field is added above the cache line without a decision about it.

Not in scope

• Router-side and cross-provider concerns (a mid-session provider switch, usage normalization in the facade). Those are tracked separately.
• llama.cpp cache_prompt / KV-cache reuse for local models — related, but a different mechanism from remote prefix caching.

References

• DeepSeek — Context caching / KV cache, pricing
• OpenAI — Prompt caching (1,024-token minimum, 128-token increments)
• Codex — prompt_caching.rs test suite, #35300, #35925, #37305
• oh-my-pi — #7324, #11897
• Jan — #8638 (compaction cache reuse, closed), #8524 (system prompt vs cached tokens)

All Jan code references are pinned to main@9fd9903.