Server: multimodal sessions lose all KV reuse on a tail token mismatch — no common-prefix rewind, no disk-KV fallback, no continued disk stores

Title: Server: multimodal sessions lose all KV reuse on a tail token mismatch — no common-prefix rewind, no disk-KV fallback, no continued disk stores

Observed on: 6289c51 (current main), DeepSeek V4 Flash Vision-Exp MXFP4, --vision server mode, --batched-session 2, disk KV enabled (--kv-disk-dir, 1 TB budget).

Summary

For multimodal (vision) chat requests, KV reuse works perfectly while the conversation round-trips exactly (multimodal live kv hit identity=fingerprint-match, sub-second turns on a ~335k-token prompt). But as soon as one request diverges from the live session by even a handful of tokens at the tail, the server falls back to a full cold prefill from token 0 — three reuse mechanisms that exist for text sessions are all unavailable or inert for multimodal ones:

1. No common-prefix rewind on live token-mismatch. The live-prefix check computes the common prefix length but only accepts it on an exact full match:

   复制

   ds4-server: live kv cache miss live=340263 prompt=340551 common=340144 vision=match reason=token-mismatch
   ds4-server: chat ctx=0..340551:340551 TOOLS prompt start

   99.96% of the prompt (340,144 / 340,551 tokens) was identical to the live session, yet the server restarted prefill from 0. The GLM path right above it does exactly this kind of rewind (rewound GLM live prefix from %d to %d, memory-rewind cache source). The DS4 path is all-or-nothing:

   c复制

   cached = common == old_pos && j->req.prompt.len >= old_pos ? common : 0;

2. Disk KV load is gated off for multimodal requests (ds4_server.c, request path around the live-miss fallback):

   c复制

   if (!multimodal && s->kv.enabled && cached == 0 && old_pos >= s->kv.opt.min_tokens) {
       kv_cache_store_current(s, slot, "evict");
   }
   if (!multimodal && cached == 0) {
       disk_cached = kv_cache_try_load(...);
   }

   So a vision session that misses the live cache never consults the disk store at all, even when a matching checkpoint exists.

3. No continued disk checkpoints are written during multimodal cold prefills. On c0a6119 the same workload logged kv cache stored tokens=... reason=continued every ~30k tokens during a long prefill. On 6289c51, three full ~330k-token prefills (18 minutes each) wrote zero checkpoints — so even if (2) were reachable, there would be nothing fresh to load. (The text-prefix memory path did resume one later turn at 100,352 of 346,533 tokens, so partial reuse is not entirely dead, but the deep disk checkpoints are never written.)

Trigger

A coding-agent client (OpenAI chat-completions, tools, one screenshot per turn) occasionally re-serializes the previous assistant reply slightly differently than the server's generated token stream (thinking/DSML round-trip). The divergence lands inside the previous reply's tail (e.g. 119 tokens before the end of a 340,263-token history). With text sessions this costs nothing; with a vision session every such turn is a full ~330k prefill ≈ 18 minutes on an M3 Ultra.

Suggested fixes

• On live token-mismatch with vision=match, rewind the DS4 session to common (same approach as the existing GLM memory-rewind path) instead of discarding the whole session.
• Allow disk-KV load/store for multimodal sessions (store text-prefix KV even if image embeddings can't be serialized; the embedding cache already covers re-encoding cheaply).

Sanitized log excerpt

21:47:24 multimodal live kv hit images=1 cached=334535 prompt=334603 identity=fingerprint-match
21:47:25 chat ctx=334535..334603:68 TOOLS prompt done 1.041s        # ~1 s turns while round-trip is exact
...
21:55:10 live kv cache miss live=340263 prompt=340551 common=340144 vision=match reason=token-mismatch
21:55:10 chat ctx=0..340551:340551 TOOLS prompt start               # full 18-min prefill, zero continued stores

(Happy to provide longer excerpts privately if useful.)