Built-in file_based KV store loses acknowledged writes on graceful shutdown

Summary

With the built-in KV store on store_method: file_based, set returns as soon as the in-memory map is updated. The write reaches disk on the next tick of the save loop (save_interval_ms, default 5000). Nothing writes the pending scopes on shutdown, so when the engine exits after a SIGTERM and before the next tick, every write acknowledged since the last tick is lost, and the client's stop still reports success.

Environment

• iii v0.11.2, the binary agentmemory 0.9.29's CLI installs (the version it pins)
• iii-state with the kv adapter, store_method: file_based, no save_interval_ms set
• Linux x86-64, Debian 13, run under systemd
• Client: agentmemory 0.9.29, which writes through state::set from the Node SDK and SIGTERMs the engine as soon as its worker exits

Reproduction

I reproduced this through agentmemory's stop path. The steps below describe the same sequence for any client.

1. Start the engine with the config above and a client worker.
2. Call state::set for a new key and wait for the acknowledgement.
3. SIGTERM the engine so that it exits before the next save tick. The tick is phase-locked to engine start (below), so a stop about 1 s after a write that lands early in the tick hits this.
4. Read the scope's .bin file under file_path. The key is missing. The file is unchanged since that scope was last persisted, or absent for a scope that was never persisted.

In the losing run the client's memory write was acknowledged at 19:38:08.368. Its shutdown save returned at 19:38:08.537, and the engine was gone by 19:38:09.33, less than 0.8 s after its SIGTERM and inside the CLI's 3 s wait before a SIGKILL. While stopped, the memory scope was unchanged, no new index scope files existed, and the index manifest was unchanged. That run's writes came about 1.2 to 1.3 s into the tick, about 3.7 s before the next one.

Four more attempts landed. My loop waited for the health check and then a fixed 8 s after each restart before writing, which put the last write of each shutdown save less than 250 ms before a tick. Across those boots the persisted files carry mtimes 10.003 to 10.004 s after engine start as recorded in the service journal, and a scope written every 30 s landed 185.004 s after start, which is 37 ticks of 5.000 s. Whether a stop loses data depends on where the write falls in the 5 s window.

Cause, from source at iii/v0.11.2 (commit 2b445957)

• BuiltinKvStore::set inserts into the map, marks the scope DirtyOp::Upsert, and returns. There is no file I/O on this path.
• BuiltinKvStore::new spawns save_loop, a tokio::time::interval that drains dirty on each tick, writes <scope>.bin.tmp, and renames it into place.
• Per serve(), the engine runs an awaited destroy() on every worker during shutdown. BuiltinKvStoreAdapter::destroy() in engine/src/workers/state/adapters/kv_store.rs is Ok(()), so that pass never reaches the pending writes. The save loop's JoinHandle is marked dead_code with the reason "Going to be used in the future for graceful shutdown".

Current main (bb2ee50)

• save_loop now has a watch::Receiver stop signal. A reconfigure hands the same dirty map to the replacement loop, so no pending write is stranded. When the sender is dropped with the store, the loop breaks without a final drain.
• The stream worker's BuiltinKvStoreAdapter in engine/src/workers/stream/adapters/kv_store.rs, which wraps its own BuiltinKvStore, has destroy() returning Ok(()).
• I could not find where the state worker's KV adapter lives on main, so I have not confirmed the state path there.

Suggested fix

Drain dirty and persist once more when the save loop stops and when the adapter is destroyed, and join the task before destroy() returns. A state::flush function would also let a client wait for durability on purpose. The iii-sdk 0.11.x state interface offers get, set, delete, list, and update, and none of them can do that. I have not checked the current SDK.

Workaround

In my deployment the client's worker now waits at shutdown until each scope it wrote is on disk with the content the engine holds, or until an 8 s deadline passes. It lists the scope, compares it with the parsed .bin, and then exits, and the CLI SIGTERMs the engine after that. With no writes arriving during shutdown, a confirmed stop takes at most one tick plus about a second. A write the worker accepts after its scope is confirmed, or to a scope first written during the wait, is not covered. From the code, lowering save_interval_ms should shrink the window without closing it; I did not test that.

Also noticed

persist_index_to_disk calls tokio::fs::write and tokio::fs::rename with no sync_all at the tag or on main, so a renamed file can still be lost to a power failure. That is a crash-durability gap, separate from the shutdown loss above.

Downstream report: rohitg00/agentmemory#1335.