Shutdown index save is lost to a container teardown race, and boot only reconciles a completely empty index

Summary

Two defects that compound. Together they can leave everything indexed since the last restart permanently unsearchable, while the data itself is intact in KV and every exit code says success.

1. The shutdown save is issued and acknowledged, then lost in transit. save() completes and process.exit(0) is reached ~0.3 s later, while the iii engine needs ~2 s to flush its file-based KV to .bin. Under docker stop, the engine is a child of PID 1 and is SIGKILLed along with the container before the flush lands.
2. Boot only reconciles a completely empty index. rebuildIndex(kv) runs solely inside if (idx.size === 0). An index that loads even one stale entry has size > 0, so the rebuild is suppressed and nothing repairs the delta.

Neither is visible: exit code 0, no error in the log, and the search API returns confident results from an index that is missing recent records.

Reproduced on a clean install

Stock @agentmemory/[email protected] on node:22-bookworm-slim with the official iii v0.11.2 binary. Fresh empty volume. No secret, no EMBEDDING_PROVIDER, no LLM key, no slots, no consolidation, no graph extraction — the log confirms No LLM provider key set — running zero-LLM (BM25 + on-device embeddings).

1. Store a memory                → mem_…91ab656701eb at 17:37:20
   Search it                     → 1 result, score 2.9124
2. Shard generation before stop  → mem%3A…%3Aidx_mtn8i4m7_ffc91ded8556%3A00000.bin, mtime 17:35
3. docker stop -t 60             → 0.3 s wall, docker exit 0, container ExitCode 0
                                    log ends at "[agentmemory] Shutting down..."
4. Shard generation after stop   → IDENTICAL, mtime still 17:35

saveShardedIndex mints a new generation on every call and has no skip-if-unchanged path, so an unchanged generation is proof the write did not land.

The save works — it is a teardown race

Running the worker as a non-PID-1 process so the engine outlives it, then sending SIGTERM to the worker alone:

gen BEFORE kill : …idx_mtn8nqb5_f1dfe47fd6b2:00000.bin
worker pid      : 130
NEW GENERATION after ~2 s : …idx_mtn8op40_6301bd8505ad:00000.bin

A/B with one variable, two runs each:

The process tree is PID 1 = node agentmemory with iii as its child. docker stop signals PID 1 only. So the write is lost in transit, not skipped — which is why no error surfaces anywhere.

The reconcile gate

rebuildIndex(kv) fires only under if (idx.size === 0) (dist/src-G7yt8gGm.mjs:3113). It has no entry cap, no time limit and no batch ceiling — it is simply gated on the index being entirely empty.

The consequence is that reconciliation is all-or-nothing. A partial gap is exactly the case it cannot repair, because a partially-stale index still has size > 0. So the state produced by defect 1 is the one state defect 2 will not fix, and it persists across every subsequent restart.

The only recovery we found is deleting the persisted shards to force size === 0:

docker stop -t 60 <container>
docker run --rm -v <volume>:/d alpine sh -c 'rm -f /d/state_store.db/mem%3Aindex%3Abm25*.bin'
docker start <container>

On the affected store this took a sample of observations from 0/4 retrievable to 4/4, writing a new generation. Worth noting that this works because it forces the empty-index precondition, not because it clears corruption.

Why the loss window is a whole session

DEBOUNCE_MS = 5e3 appears never to fire during runtime. Four memories plus 60 s of idling produced no BM25 shard at all; every generation observed was minted at boot or at a shutdown that was allowed to complete.

If that is right, boot and shutdown are the only durable write paths — and shutdown is the racing one. That is what turns this from "you lose the last five seconds" into "you lose everything since the last restart".

Two things that make it silent

• Exit code 0 is not evidence. The shutdown handler ends in process.exit(0) regardless of whether the flush landed.
• Failed to save index on shutdown can never fire. save() wraps its body in try/catch and routes errors to logFailure, so it never rejects and the .catch() at the shutdown call site is unreachable. Not the cause here, but it means a genuine save failure would also be invisible.

Suggested fixes

1. Await the engine flush before exiting, or give the engine its own SIGTERM and a grace window, rather than reaching process.exit(0) 0.3 s after issuing the save.
2. Let boot reconcile a partial index, not only an empty one — compare the index against KV rather than gating on size === 0. A partial gap is the realistic failure mode and currently the one case that never repairs.
3. Make the debounce actually persist during runtime, so a lost shutdown costs seconds rather than a whole session.
4. Remove or fix the unreachable .catch, so save failures are observable.

Item 1 alone would close the common case.

One thing we could not explain

In one clean-room run a stale shard (mtime 17:35) was present, no new generation was written on restart, and yet the memory stored at 17:37:20 was still found afterwards at the identical pre-stop score. Under the gate described above it should have been missing. Either the shard load silently failed — yielding size === 0 and a rebuild that then did not persist — or there is a re-index path we did not trace. Stating it rather than leaving it out, since it may indicate a third behaviour.

Environment

• agentmemory 0.9.29, iii engine v0.11.2
• Clean-room repro: node:22-bookworm-slim, stock npm package, fresh volume, default config
• Originally observed on a 112 MB store (~99% graph) with LLM compression and local embeddings enabled, then reproduced with all of that removed

Related

#1180 (index saves scheduled only at boot) describes an adjacent symptom; this report identifies the teardown race and the size === 0 gate as the mechanism. #1179 (rebuild only runs on an empty index) is the same gate seen from the embeddings side.