Shutdown checkpoint is never found on restart when the conversation began after another live conversation in the same process
Build: main 9139e2a, Metal, M4 Max 64 GB, Qwen3.8-Flash-Next-Q2.gguf (self-contained), --mtp -c 163840 --prefill-chunk 1024 --kv-disk-dir DIR --kv-disk-space-mb 16384. Same result on ivanfioravanti/ds4-metal 6c1e836 (identical ds4_kvstore.c), so it predates the Qwen merge and is not specific to this model's checkpoint payload.
Repro (10 minutes, two small OpenAI-compatible clients)
- Conversation A: 4 chat turns, ~35–40K tokens.
- Conversation B: 8 turns with tools, ~80K tokens. B's first request logs
live kv cache miss live=38403 prompt=11129 common=3 vision=match reason=token-mismatchthenkv cache stored tokens=38403 ... reason=evict(A goes to disk), and B is prefilled. SIGTERMthe server. Log:persisting resident KV cache before shutdown slot=0 tokens=79188→kv cache stored tokens=79188 ... reason=shutdown key=token-text. The file is on disk; its header SHA matches its text; the text starts with B's system prompt.- Restart with the same flags and
--trace; resend B's exact history plus one user turn (~90K tokens).
Result: full re-prefill. Trace for that request:
memory_miss_reason: no-live-checkpoint
cache_source: none
cached_tokens: 0
disk_cached_tokens: 0No kv cache hit and no kv cache skipped line; the lookup found nothing, not even B's smaller continued checkpoints.
What does and does not matter (all tested)
| Variant | Restores after restart? |
|---|---|
| B alone in the process (no A), empty cache dir | yes (cache_source: disk-text, 80,091 tokens, ~0.3 s) |
B alone, cache dir at budget (--kv-disk-space-mb 3072) |
yes |
| A (with tools) then B (with tools) | no |
| A without tools, then B with tools | no |
| A with tools, then B without tools | no |
So tool calls, cache pressure and the eviction order of #444 are ruled out. The one condition that flips the result is whether B started cold (fresh process, live position 0) or warm (another conversation was live; B's first request matched a few common tokens of it and the rest was re-prefilled). In every warm case B's first continued checkpoint lands at 8,192 tokens instead of the cold case's 10,240, so B's session bookkeeping differs from the first turn on.
Where I got to in the code (pointers, not a diagnosis)
ds4_kvstore_find_text_prefixhashes the prompt prefix against every index entry (longest SHA1 match), so a miss means no stored text is a byte prefix of the replayed prompt.- The stored text comes from
slot->thinking_live.visible_text/responses_live.visible_textwhen that record is valid andlive_tokens == tokens->len(kv_cache_store_current, ds4_server.c ~11275), otherwise fromds4_kvstore_render_tokens_text(detokenised).token_text_disk_keyis set fromprompt_preserves_reasoning && !visible_continuation. - My guess is that after a warm start the stored text for B is produced on a different path than after a cold start (detokenised vs the client's rendered text, or with/without the thinking-visible handling), so its SHA1 can never equal the hash of the client's bytes. I did not verify which branch was taken; a log line naming the text source at store time would settle it in one run.
Why it matters
A daily server always has other conversations behind it (cron jobs, side requests), so in practice the live conversation is never restored after a graceful restart; at 160K that is an ~8-minute first turn every time the service restarts.
Clients used: two ~150-line Python scripts (streaming, tools); happy to attach them or run a variant with extra logging.
Source: antirez/ds4