[Bug] MP server: lookup reports full hit but restored KV context is wrong for long prefixes (breaks between 114 and 128 chunks)

Summary

In multiprocess (MP) mode, a prompt that was previously stored is looked up and reported as a full hit (lookup_requested == lookup_hit == expected tokens, per-rank h2d transfers observed), yet the restored KV context produces a greedy continuation that is unrelated to the continuation produced by the same engine's own cold computation of the same prompt (common prefix: 1 char of 2045). At shorter lengths the identical flow is byte-verbatim.

The failure is same-instance and same-process: engine computes prompt P cold (stores it), waits until stores fully settle, resets the engine-local prefix cache, then recomputes P — the second pass reports a full external hit and finishes fast (TTFT 0.74 s vs 60.3 s cold), but the output diverges from char ~1. This rules out cross-instance numerics, producer identity, and client-side effects.

Bracket (same-instance flow, verbatim vs broken):

prompt tokens	chunks	result
32 768	41	verbatim (agree 1540/1540)
65 536	83	verbatim (cross-instance as well)
90 112	114	verbatim (agree 2522/2522, hit counter exact)
100 352	128	BROKEN (agree 1/2045, hit counter reports full)

Breakpoint is inside (114, 128] chunks. We have not bisected further.

Environment

• LMCache: frozen build v0.5.6.dev7 (ge5078730), MP server (lmcache server --chunk-size 784 --separate-object-groups) + LMCacheMPConnector on vLLM engines (KV connector V1).
• vLLM: v1.5.0-based build, TP4.
• Model: Qwen3.8-27B-FP8, hybrid GDN + full attention, 784-token hybrid KV chunk (64 layers).
• Realm: L1 40 GiB host memory + fs_native L2 (write-through). Failure reproduces with the working set far below the L1 watermark — no eviction involved.

Reproduction (same-instance, no confounds)

1. Launch one TP4 engine against a fresh MP server.
2. Send prompt P (100352 frozen token ids, temperature=0, seed fixed, 512 output tokens) — cold compute, ~60 s TTFT. Wait until Stored 784 lines stabilize (128 chunks × 4 ranks).
3. POST /reset_prefix_cache (engine-local APC only).
4. Re-send the identical P.
5. Observe: lmcache_mp_lookup_hit_l1_tokens_total delta == 100352 (hit_l2 == 0), engine external_prefix_cache_hits_total delta == 100351, per-rank h2d bytes > 0 — and the output agrees with step 2's output on 1 of 2045 characters.

Control at 32768 (41 chunks): same flow, output verbatim, hit counter exact (32144).

Script: tier_diag_selfhit.py (archive available on request); raw evidence: results-tiered/diag-acceptance.json and the phase acceptance with the cross-instance variant of the same signature (results-tiered/phase-b-partial-acceptance.json, L100352/l1/rep0).

Additional observations

• h2d volume mismatch: at the 100352 hit, total h2d staging bytes were ~6.8 GiB across 4 ranks where a full 100351-token restore implies ~26 GiB at this model's ~262,144 B/token (4-rank combined). The counters say "full hit" while the transferred volume says ~quarter — suggesting the lookup counted tokens/chunks it did not actually transfer, or assembled the context from a subset.
• Off-by-one: external hits 100351 vs expected 100352 at the boundary length.
• Stores were fully settled before the lookup (line-count + object-count stability gate), so this is not an in-flight-store race visible from our side. We cannot rule out a store-side metadata/counting race that the settle window hides.
• Not eviction: the 26 GiB working set fits the 40 GiB L1 with the 0.8 watermark, and the realm object census after the store shows it resident.
• Impact: silent wrong-context serving — for any prefix ≥ the break point the engine returns fast, confidently wrong continuations. This makes tiered caching unusable beyond ~83 chunks on this build and is why our phase marked all ≥100352 rows NOT_RUN.

Ask

• Pointers to the MP lookup/reassembly path for multi-chunk prefixes (chunk hashing, hit accounting vs transfer accounting) would help us bisect further — the (114, 128] bracket already suggests a threshold near 2^7 chunks or a per-prefix chunk-count limit.
• We can run any requested bisection on this hardware within a day.