TurboQuant KV cache (4-bit) silently corrupts detail retrieval for requests that hit the prefix cache

Environment

• oMLX 0.6.4 (macOS app), macOS 26 on MacBook Air M4 (10-core, 24 GB)

• Model: Ornith-1.5-9B-MLX-4bit (Qwen3.5-9B hybrid architecture, model_type: qwen3_5; 32 layers, 8 true KV-cache layers)

• Settings: turboquant_kv_enabled=true, bits=4, skip_last=true

Summary

When TurboQuant KV is enabled, any request whose prompt hits the block-level prefix cache fails to retrieve details from the cached region of the prompt. The failure is completely silent — the model answers fluently, just wrong. Disabling TurboQuant makes the exact same cached-prefix requests pass 100%.

Reproduction

Needle-in-a-haystack @ 8k tokens, needle at 60% depth. Ask the identical question repeatedly in two modes:

• No prefix: send the exact same prompt each time → hits the prefix cache

• Unique prefix: prepend a random nonce line each time → cache miss, full re-prefill

Question: "What is the backup door password mentioned in the record?" (needle: QX-7391-ZULU)

Results (all on the same machine, same session, temperature 1.0/top_k 20)

Controlled variables already ruled out:

• Position (lost-in-the-middle): depth sweep 10/10 pass when needle falls outside the reused region

• Prompt-guided attention: neutral random prefixes (no identifier semantics) also pass 3/3

• Length: 24k-token prompts pass 2/2 (cold cache)

• Cold-cache first ask without prefix passes; only reused cache fails

The failure mode is characteristic: the model reproduces the document's generic boilerplate perfectly but cannot see specific details inside the reused segment.

Engine log signature (confirming the config was actually active, from ~/.omlx/logs/server.log):

Expected: prefix-cache reuse should be functionally equivalent to a full re-prefill, regardless of KV quantization.

Actual: with TurboQuant 4-bit, reused KV blocks lose fine-grained information, and there is no error, warning, or log entry.

Impact

This is severe for agent / RAG workloads — which are precisely the ones oMLX's tiered KV caching targets (repeated document Q&A, coding agents re-reading the same files). The user sees "the model is occasionally dumb" rather than an error, making it very hard to diagnose.

Workaround

• Disable TurboQuant (retrieval 100% with plain KV at 8k–24k on this model), or

• Force cache miss with a unique prefix per request (costs a full re-prefill)

Happy to provide the full test harness (Python) and raw results if useful.