#1053·ds4

Shutdown checkpoint is never found on restart when the conversation began after another live conversation in the same process

Author: pswaiCreated Sep 14, 2026Updated Sep 18, 2026

Build: main 9139e2a, Metal, M4 Max 64 GB, Qwen3.8-Flash-Next-Q2.gguf (self-contained), --mtp -c 163840 --prefill-chunk 1024 --kv-disk-dir DIR --kv-disk-space-mb 16384. Same result on ivanfioravanti/ds4-metal 6c1e836 (identical ds4_kvstore.c), so it predates the Qwen merge and is not specific to this model's checkpoint payload.

Repro (10 minutes, two small OpenAI-compatible clients)

  1. Conversation A: 4 chat turns, ~35–40K tokens.
  2. Conversation B: 8 turns with tools, ~80K tokens. B's first request logs live kv cache miss live=38403 prompt=11129 common=3 vision=match reason=token-mismatch then kv cache stored tokens=38403 ... reason=evict (A goes to disk), and B is prefilled.
  3. SIGTERM the server. Log: persisting resident KV cache before shutdown slot=0 tokens=79188kv cache stored tokens=79188 ... reason=shutdown key=token-text. The file is on disk; its header SHA matches its text; the text starts with B's system prompt.
  4. Restart with the same flags and --trace; resend B's exact history plus one user turn (~90K tokens).

Result: full re-prefill. Trace for that request:

memory_miss_reason: no-live-checkpoint
cache_source: none
cached_tokens: 0
disk_cached_tokens: 0

No kv cache hit and no kv cache skipped line; the lookup found nothing, not even B's smaller continued checkpoints.

What does and does not matter (all tested)

Variant Restores after restart?
B alone in the process (no A), empty cache dir yes (cache_source: disk-text, 80,091 tokens, ~0.3 s)
B alone, cache dir at budget (--kv-disk-space-mb 3072) yes
A (with tools) then B (with tools) no
A without tools, then B with tools no
A with tools, then B without tools no

So tool calls, cache pressure and the eviction order of #444 are ruled out. The one condition that flips the result is whether B started cold (fresh process, live position 0) or warm (another conversation was live; B's first request matched a few common tokens of it and the rest was re-prefilled). In every warm case B's first continued checkpoint lands at 8,192 tokens instead of the cold case's 10,240, so B's session bookkeeping differs from the first turn on.

Where I got to in the code (pointers, not a diagnosis)

  • ds4_kvstore_find_text_prefix hashes the prompt prefix against every index entry (longest SHA1 match), so a miss means no stored text is a byte prefix of the replayed prompt.
  • The stored text comes from slot->thinking_live.visible_text / responses_live.visible_text when that record is valid and live_tokens == tokens->len (kv_cache_store_current, ds4_server.c ~11275), otherwise from ds4_kvstore_render_tokens_text (detokenised). token_text_disk_key is set from prompt_preserves_reasoning && !visible_continuation.
  • My guess is that after a warm start the stored text for B is produced on a different path than after a cold start (detokenised vs the client's rendered text, or with/without the thinking-visible handling), so its SHA1 can never equal the hash of the client's bytes. I did not verify which branch was taken; a log line naming the text source at store time would settle it in one run.

Why it matters

A daily server always has other conversations behind it (cron jobs, side requests), so in practice the live conversation is never restored after a graceful restart; at 160K that is an ~8-minute first turn every time the service restarts.

Clients used: two ~150-line Python scripts (streaming, tools); happy to attach them or run a variant with extra logging.