Semantic search silently stops saving new memories once the store passes a size set by the embedding width, and a restart does not rebuild them

Version: @agentmemory/agentmemory 0.9.29 · Node v22.23.2 (MAX_STRING_LENGTH = 536,870,888) · Linux x64 · EMBEDDING_PROVIDER=openai against a local OpenAI-compatible server, OPENAI_EMBEDDING_DIMENSIONS=4096

All line numbers are dist/index.mjs of the published 0.9.29 tarball.

Summary

Once a store grows past a size that depends on the embedding model (about 24,000 memories at 4096 dimensions, about 65,000 at 1536), agentmemory silently stops saving anything new for semantic search. Older memories still come back and keyword search keeps working, so recall looks merely weak rather than broken. Nothing in the logs, the health endpoint or the diagnose tool says a limit was hit, and a restart does not repair it: the store comes back with the last set of memories that fit and never rebuilds the rest. Every memory recorded after that point is invisible to semantic recall until an operator prunes the store by hand.

Mechanism

The vector index is serialised into one string before sharding, so at 4096 dims every save throws RangeError: Invalid string length past 24,471 entries. BM25 keeps saving. On the next restart the store loads the last vector generation that fit, or nothing if the shards were ever moved, and because both rebuild gates test the BM25 index, nothing rebuilds the vector index. Vector search answers from the last snapshot that saved.

Open PR #1258 fixes the serialisation. It does not fix the gates, and it does not fix the reason the index reached the ceiling. Those are the changes requested here.

Earlier filings against the same code path: #309, #762, #764 and #1258. None names MAX_STRING_LENGTH or Invalid string length.

The whole index is one string

VectorIndex.serialize() (1656) base64s every vector and JSON.stringifys the array. saveShardedIndex (2978) slices that finished string at shardChars offsets, so shard size cannot help, and shardChars is not configurable in production anyway: new IndexPersistence(kv, bm25Index, vectorIndex) (22846) passes no options.

A row is ["<obsId>",{"embedding":"<base64>","sessionId":"<sessionId>"}], 36 fixed chars plus 4 * ceil(dim * 4 / 3) base64 chars plus the ids, and the document is n * (row + 1) + 1, which gives

ceiling = floor((MAX_STRING_LENGTH - 1) / (base64_chars + 36 + len(obsId) + len(sessionId) + 1))

Id lengths are as in the reproduction below. This store's real ids give 24,463. Dimension moves the ceiling by an order of magnitude, id length by a fraction of a percent.

BM25 saves anyway, and the last good vector generation stays on disk

save() (2930) wraps both streams in one try with BM25 first (2936-2937), so the BM25 manifest advances on every pass while the vector save throws. Previous-generation cleanup (3038) only runs after a completed save, so the last vector generation that fit stays referenced by a valid manifest and loads cleanly on restart.

Nothing rebuilds it

Both rebuild gates test BM25: bm25Index.size === 0 at boot (22872) and idx.size === 0 in the search handler (3619), where idx is getSearchIndex(), the SearchIndex. BM25 is healthy, so neither fires. This does not depend on the format: any vector-only failure (a KV timeout, a torn shard) leaves the same unrecoverable state through the same gates.

The read path has the same ceiling

loadManifestData (3111) ends in chunks.join("") (3153), so an index over the ceiling cannot be read even if something wrote it. That failure is loud: the RangeError reaches the .catch at 22848, which returns null, discards BM25 too, and triggers a full rebuild every boot. A patch that fixed only the write side would make a store reachable that cannot be loaded.

Reproduction

Extract VectorIndex and float32ToBase64 verbatim from 1567-1686 and drive serialize() with one shared Float32Array, so the measurement is string cost alone:

MAX_STRING_LENGTH       536870888   (buffer.constants, Node v22.23.2 x64)
base64 chars per vector 21848
JSON chars per row      21938 + 1   (obsId 25 + sessionId 29 + 36 fixed + comma)
serialize(24471) -> ok, length 536869270
serialize(24472) -> RangeError: Invalid string length

The formula reproduces both numbers exactly, and at 1536 dims predicts 64,815, where serialize(64815) succeeds and serialize(64816) throws. Against a real store the process may die of heap exhaustion first, roughly 401 MB of Float32Array beside 536 MB of base64, which is the same bug with a different error.

In production: a 4096-dim provider over a 30,327-entry corpus. Every debounced save threw, logged once a minute through logFailure (2960):

[agentmemory] warn index persistence: failed to save BM25/vector index {"message":"Invalid string length"}

48 BM25 shards written, zero vector shards. After pruning the corpus offline to 22,182 entries the same build saves cleanly: 244 vector shards, 486,806,083 chars against the 536,870,888 ceiling.

Expected: the vector index persists at any size the process can hold in memory, or the save failure triggers a rebuild on the next boot. Actual: every save past 24,471 entries fails, the next boot loads the last generation that fit, and neither rebuild gate fires.

What the operator sees

• The common outcome is stale, and only a store whose shards were moved comes up empty. Vector search keeps returning results from the snapshot taken when the corpus crossed the ceiling.
• The boot log looks healthy. In the stale case loaded.vector.size > 0, so 22869 prints Loaded persisted vector index (N vectors) next to the BM25 line. The audit trail shows it: index_persist records (safeAudit, 3047) show BM25 manifest_publish entries advancing while vector entries stop.
• No health surface reports it. collectHealth (15650) reports memory, CPU, workers and a KV probe, and nothing about index sizes or the last successful save. mem::diagnose's ALL_CATEGORIES (12140) has no index category.

Severity

Any operator whose index reaches the ceiling for their dimension. Exposure requires OPENAI_EMBEDDING_DIMENSIONS (or the OpenRouter equivalent) to be set, because without it resolveDimensions (939) guesses 1536 for any untabled model and the provider guard (withDimensionGuard, 1249) rejects every vector before the index can fill. That discard is filed separately. At 1536 dims the ceiling is 64,815 entries, which a long-lived store reaches.

Covered by #1258, no further change requested there

PR #1258 ("perf(state): write only the vector buckets that changed") is framed as a write-amplification fix, so it is not recorded anywhere that it also closes this ceiling. Verified against the diff:

• serializeBuckets is a generator that stringifies 256 buckets separately, so no whole-index string is ever built and the write ceiling is gone.
• loadVectorBuckets joins per bucket, so the read ceiling is gone.
• save() gets one try per stream with vectors first, so a vector failure is no longer hidden by a BM25 success.

Each bucket is still one JSON.stringify and one join, so the same ceiling exists per bucket at 256x the distance (about 6.26M entries at 4096 dims). At that distance a comment naming the per-bucket ceiling is enough.

What needs to change

#1258 fixes the ceiling. The rest, in the order it should land:

1. Rebuild once on a v1 manifest. loadVectorBuckets returns null for a v1 manifest before vectorLoadRejected is computed, so load() falls through to the v1 reader, the stale generation loads cleanly, vectorRejected stays false, BM25 is nonzero, and the new gate bm25Index.size === 0 || vectorRejected never fires. Saves then succeed and the index heals forward from that point, so everything embedded between the ceiling and the upgrade is lost silently. Set a vectorLegacyFormat flag on the v1 fall-through and rebuild on it. It fires once per store. The same rebuild is the place to delete the v1 shard keys, which are otherwise never reclaimed because the first v2 save writes previousV2 = null.
2. Make rebuild and save() mutually exclusive, and share the guard. rebuildIndex (3557-3560) clears both indexes synchronously and refills over many awaits, while flushIndexSave (3401) is reachable from nine call sites with no interlock. A save landing mid-rebuild publishes a partial index as a completed generation and GCs the previous one (3038). A restart at that moment leaves a truncated BM25 index that satisfies bm25Index.size > 0 forever. withKeyedLock (3193) already exists, or rebuild into fresh objects and swap with restoreFrom. Route the boot gate (22872) through rebuildPromise (3379) as well. Only the search handler sets it today (3620), so a search during the boot rebuild starts a second one. This is a live data-loss path independent of the ceiling, and serializeBuckets yielding across turns makes it routine.
3. Key rebuilds on the vector index. Both gates (22872, 3619) test the BM25 index, so any vector-only failure is permanent. vectorIndex.size === 0 does not work, as #1258's comment says: rebuildIndex returns a BM25 count, and a store whose embeds are all rejected would re-embed its whole corpus on every boot. Rebuild when vectorIndex.size is far below bm25Index.size with a provider active, and record the attempt in a KV key outside KV.bm25Index (1487, the scope operators move aside) at the moment the rebuild fires, recording provider identity, width and the post-rebuild vector/BM25 ratio. Re-fire only when the ratio degrades or the provider changes. Suppress on vectorLoadIncomplete, which #1258 sets when a transient manifest read returns an empty index. This depends on fix 2.
4. Make mem::evict remove from both indexes. It removes from neither (6423-6601). Every other site pairs getSearchIndex().remove with vectorIndexRemove (6323-6324, 9505-9506). Its stale-session branch also deletes the session row (6465) but not its observations, which rebuildIndex can then never reach (3569-3575). mem::forget (6328-6341) is the model. Removing from the vector side alone would open a bm25 − vector gap equal to the evicted count and false-positive fix 3, so pair them.
5. Report vector size, BM25 size and last successful save per stream in collectHealth. The dimension issue asks for the same. collectHealth is inside registerHealthMonitor(sdk, kv) (15643) and closes over neither index nor IndexPersistence, constructed one line later (22846). vectorIndex (3373) and indexPersistence (3397) have setters and no getters, and no last-save field exists. Estimate proximity to the ceiling from the formula above. serialize().length is the throwing operation and cannot be the measurement. Suppress while a rebuild is in flight.

One caution for whatever lands

Three hard-coded v === 1 checks are not symmetric: loadManifestData (3111) returns null on an unknown version, the previous-generation GC (3038) is gated on previous?.v === 1 and orphans every superseded generation if missed, and isManifestPublished (3072) returns false for v !== 1, which sends saveShardedIndex down deleteShards(shards, "manifest_publish_rollback") (3034) against shards a published manifest may already point at. Any format change needs all three. BM25 has the identical whole-string ceiling (SearchIndex.serialize, 2772), near 195,000 records at the sizes measured here, and its payload is not a flat row list, so it needs its own decomposition.

Related

Filed separately: resolveDimensions falls back to 1536 for any model outside a three-entry table and the provider guard then rejects every embedding a wider model returns. Setting the override to escape that bug lets the index fill and reach this one. A store still in that bug, with every embed rejected, is the case fix 3's marker guards against.

#1115 (orphaned shard generations) is the same saveShardedIndex from the other side: generations that a failed publish or a mid-save death leave on disk with no reconcile to reclaim them. A comment there follows the version skew named in the caution above through to a route that deletes live shards or strands a whole set.

#1223 (the health endpoint's memory field): the one number collectHealth already reports is wrong. A comment there covers it.