Vector storage: a model or dimension change must fail closed, and lightrag-rebuild-vdb must be able to recover on every backend

Summary

Two related defects across every vector backend, on the same trigger — the operator changes the embedding model or dimension:

• A. The system fails OPEN. Depending on the backend, LightRAG starts and serves either an empty vector index or one holding vectors from a different embedding space. Retrieval silently returns nothing, or nonsense. No backend refuses to start on a same-dimension model swap.
• B. The sanctioned recovery tool cannot run. lightrag-rebuild-vdb is the documented answer to "I changed the embedding model" (the tool's own docstring says so). On the backends that do fail closed, the tool aborts during its own startup — it initializes every storage through the server's path before it can drop anything. The operator must delete the container out of band (OpenSearch API, DROP TABLE, rm the file) before the recovery tool will start.

The two are coupled: A cannot be fixed without B. Failing closed is only acceptable if a working way out exists.

Both goals, stated once:

1. A model or dimension change must fail closed — refuse to serve, with a message naming what changed and what to run.
2. Every vector backend must be recoverable with lightrag-rebuild-vdb after that change, with no out-of-band step.

Design decisions below were refined during review of PR 1 (#3986). The full, current reasoning — including everything already rejected — lives in docs/design/VectorSpaceProvenance.md. The Design decisions, Breakdown and Open question sections of this issue have been updated to match; the Current state table and the Salvage from #3966 section are unchanged.

Current state

backend	dimension change	same-dimension model change	lightrag-rebuild-vdb after the change
Nano	AssertionError from NanoVectorDB.__init__ — fail closed, unhelpful message	silently reuses the old vectors	no — raises in NanoVectorDBStorage.__post_init__ (nano_vector_db_impl.py:151), i.e. at construction, before initialize()
Faiss	ValueError on index load — fail closed	silently reuses the old vectors	no — _load_faiss_index() is called from __post_init__ (faiss_impl.py:210), so it too raises at construction (faiss_impl.py:1323)
MongoDB	ValueError from the dimension guard — fail closed	silently reuses the old vectors	no, and worse: drop() does delete_many({}) and then create_vector_index_if_not_exists(), which re-reads the surviving Atlas search index definition and raises again. except PyMongoError does not catch it. Unrecoverable even if the tool reached drop()
OpenSearch	ValueError from the dimension guard — fail closed	silently reuses the old vectors	no — initialize() raises before _flush_lock is set, so drop() then dies on async with None. drop() itself would recover (it deletes the whole index and recreates it)
Milvus	new suffixed collection, created empty — fail open	new suffixed collection, created empty — fail open	yes
Qdrant	new suffixed collection; a legacy collection of another dimension raises DataMigrationError (qdrant_impl.py:276)	new suffixed collection, created empty — fail open	not on the legacy path — the migration raises during initialize()
PostgreSQL	new suffixed table; a legacy table of another dimension raises DataMigrationError (postgres_impl.py:4215)	new suffixed table, created empty — fail open	not on the legacy path — same shape as Qdrant

Two distinct fail-open shapes, and the second is worse than the first:

• Suffixed backends (Milvus / Qdrant / PostgreSQL) land on a new, empty container. Queries return nothing.
• Un-suffixed backends (Nano / Faiss / MongoDB / OpenSearch) reuse the same container on a same-dimension swap. Queries return confidently wrong neighbours. Nothing detects this anywhere today.

Design decisions

Failing closed is the requirement; the model-isolation suffix is not the mechanism. The suffix makes a model change land on a fresh empty container and the service start normally — that is the fail-open behaviour above. #3966 proposed adopting it for OpenSearch and explicitly chose fail-open on both the dimension-change and model-change branches (logger.warning(...), return None), justified by "raising would reproduce the wedge this PR removes". Once defect B is fixed, that justification is gone. #3966 will be closed and reworked against this issue.

A typed exception is load-bearing. Today these gates raise bare ValueError, and Nano raises AssertionError. A tool that tolerates the refusal by catching Exception would swallow a cluster outage, a bad credential or a corrupt file as "just a model change" and drop data on a false positive. Nothing can tolerate this condition safely until it is distinguishable. Landed as VectorSpaceMismatchError in PR 1; deliberately not folded into DataMigrationError, since nothing is being migrated.

Source storages keep the server-identical init path. The scope of "the tool must not run the server's startup" is the vector targets only. The graph store and text_chunks are the authoritative sources of the rebuild; skipping their migrations would rebuild vectors from a half-migrated source. The tool's docstring ("Run it like a server startup, not like a pure read") stays true for them.

The model name must be recorded, not inferred — but only where the container name does not already carry it. A dimension is not an identity: two models can share 1024 dimensions and have unrelated vector spaces. Sorting the backends by how they isolate decides who needs a marker:

backend	workspace isolation	model isolation	name answers "which model?"
Nano, FAISS	subdirectory	none	❌
MongoDB, OpenSearch	collection / index name prefix	none	❌
Milvus	collection name prefix	collection name suffix	✅
Qdrant	point-id salting + workspace_id payload filter	collection name suffix	✅
PostgreSQL	workspace column	table name suffix	✅

So Nano, FAISS, MongoDB and OpenSearch record a marker — they are exactly the four where a same-dimension swap reuses the same container. Milvus, Qdrant and PostgreSQL record nothing: a model change already lands in a different container by construction, and a second copy of a fact the name carries only drifts. Qdrant's collection name carries no workspace, which is not a gap — its collections are multi-tenant by design (salted point ids plus a tenant-indexed payload filter), so a collection-level marker could not express "this workspace's model" even in principle, and does not need to.

The marker never lives in the data plane. A record carrying a vector enters the ANN index, and anything in the ANN index can be returned by a search — a marker recalled as a search hit is a fabricated chunk entering an LLM's context. Homes: OpenSearch → index mapping _meta; MongoDB → the collection's JSON Schema validator description; FAISS → a reserved key in .meta.json; Nano → additional_data in the vdb JSON. Already rejected: a marker point on Milvus/Qdrant (both require a vector; Milvus queries carry no filter at all), a shared sidecar collection (a cross-workspace container in an otherwise per-workspace design, and unnecessary), and a marker document inside the Mongo vector collection (not recallable, but it owes every future full-collection scan an exclusion).

Adoption of an unmarked container carries evidence. Indices predating this record nothing, and absent evidence never refuses — otherwise every pre-upgrade container is refused on the first start. But blind adoption is worse: an operator who upgrades and switches to a same-dimension model in one step would get the new model's name stamped onto the old model's vectors, recorded permanently, after which the gate can never fire. So: an empty container is adopted unconditionally; a non-empty one only after a round-trip probe — re-embed one record's stored content and compare it to its stored vector (same model ≈ 1.0, a different model typically 0.0–0.5).

The probe runs one layer up, in LightRAG.initialize_storages(): BaseVectorStorage has no enumeration API, so a backend cannot sample its own records, while the layer above already has the graph to supply ids. One probe settles all three storages, and costs nothing once a marker is written. Its failure semantics are the load-bearing part: only a probe that ran and returned a negative verdict may refuse. An embedder outage, a timeout (explicit — the providers retry with exponential backoff), no usable sample, or a failed marker write all fall back to the pre-upgrade behaviour: serve, stay unmarked, retry next start.

Breakdown

One PR per reviewable unit. The exception ships with the tool PR; each backend PR then raises it and makes its own storage recoverable.

• PR 1 — lightrag-rebuild-vdb + the shared exception (#3986). Add the typed vector-space-mismatch exception. In the tool: keep graph / text_chunks on the current init path and abort on their failure as today; tolerate only the typed refusal from the three VDB targets; open the rebuild (options 2/3/4) with drop() on a refused VDB; make the consistency check (option 1) report the incompatibility instead of emitting a false "everything is missing" report. No behaviour change until a backend PR lands.
• PR 2 — OpenSearch. Raise the typed exception; set the client and _flush_lock before the compatibility gate so a refused instance stays drop-capable; record the embedding model in the index _meta and refuse on a mismatch. _claim_index_for_workspace already performs a merging put_mapping, so a marker write can ride along. Also closes a bug on main: the loser of an indices.create race attaches to an index it did not build and validates ownership only. Replaces #3966.
• PR 3 — MongoDB. Same gate; marker in the collection's JSON Schema validator description. drop() must actually drop and recreate the Atlas search index on a dimension or model change — the FAILED-index branch already has drop_search_index + _wait_for_search_index_absent to reuse — and rewrite the validator description, since delete_many({}) leaves it behind and a stale one would make the next initialize() refuse again. A collMod permission failure degrades to unmarked, never to a refusal.
• PR 4 — Faiss and PR 5 — Nano. Typed exception instead of ValueError / AssertionError, plus the part specific to both: the refusal happens inside __post_init__ (faiss_impl.py:210 -> 1304; nano_vector_db_impl.py:151), so the tool cannot even construct the storage, let alone drop it. The check has to move to a point where the object survives its own refusal and stays droppable — the tool resets the index and metadata files through it. Marker in .meta.json / additional_data.
• PR 6 — Milvus, PR 7 — Qdrant, PR 8 — PostgreSQL. No marker (see Design decisions). What remains: replace the legacy-path DataMigrationError with the typed refusal so the tool can tolerate it rather than being wedged during initialize(), and make a refused instance drop-capable — Qdrant assigns _flush_lock after its init block, the same shape as the OpenSearch bug. Their fail-open is closed by the gate below, not here.
• PR 9 — the gate and the adoption probe, in LightRAG.initialize_storages(): refuse when the vector store is empty while the graph is not, and adopt unmarked containers per Design decisions.

Settled: where the suffixed backends' fail-open is detected

(Was an open question. Settled during PR 1 review — full reasoning in docs/design/VectorSpaceProvenance.md.)

For Milvus / Qdrant / PostgreSQL, a marker can never fire: the new suffixed container is genuinely theirs and correctly named. The choice was between enumerating sibling containers per backend and the cross-storage question — the vector store is empty while the graph is not.

The cross-storage form wins, one layer up. The deciding argument is that it self-clears: once the rebuild populates the container the question answers itself. Sibling enumeration does not — the stale sibling outlives a successful rebuild, so the signal would have to be cancelled by something else, and drop() cannot cancel it, because a dropped-and-re-provisioned container is indistinguishable from a freshly created one. It is also one implementation instead of three, and it catches what no marker can: a deleted vector file, a container emptied out of band, an interrupted rebuild.

Accepted residues

Each has a recovery path; all are recorded in the design doc.

• Fold collision on Milvus / Qdrant / PostgreSQL: the suffix lowercases and folds punctuation, so text-embedding-3-large and text_embedding_3_large share a container. A harmful collision needs two genuinely different models whose names differ only that way and which share a dimension; in practice such pairs are the same model spelled differently.
• No EMBEDDING_MODEL configured: the suffixed backends fall back to an un-suffixed container carrying no model information, so those deployments get no gate, exactly as today. Recovery: set it, then run lightrag-rebuild-vdb.
• Embedder unavailable during an adopting start that coincides with a same-dimension swap: the probe cannot run, so the instance serves wrong results until a later start probes successfully. Strictly better than today, where nothing ever detects it.

Out of scope

• Re-embedding inside a storage backend. No backend does it, and lightrag-rebuild-vdb rebuilding from the authoritative graph is the supported path.
• Keeping a previous model's vectors around for cheap rollback. That is what the suffix bought, and it is bought with the fail-open this issue removes. If it is wanted later it needs its own design, on top of a working fail-closed gate.

---

Salvage from #3966

#3966 is closed because its mechanism — the model-isolation suffix — is the fail-open this issue removes. Several things it built are independent of that mechanism and should be carried into the PRs below rather than rediscovered. Branch feat/opensearch-model-isolation-suffix, commits 1439ec3b5..7daaa4f50; several of these took multiple review rounds to get right.

Carry into PR 2 (OpenSearch) mostly as-is

_assert_index_is_usable(mapping) — one choke point for "can we read this index". The PR's most reusable finding is that OpenSearch has three places that attach to an index it did not just create, and on main they check different things:

attach point	dimension	model
_create_knn_index_if_not_exists, exists() branch	✅	❌
_recheck_index_presence (read-path readiness probe)	✅	❌
_create_knn_index_if_not_exists, resource_already_exists_exception branch	❌	❌

The third row is a bug on main independent of everything in this issue: when two workers start together, the loser of the indices.create race attaches to an index it did not build and validates ownership only. A deployment whose name folds onto the same index can mark itself ready against vectors of another dimension entirely. The PR closes it by validating after _claim_index_for_workspace on that branch too (lost_race). Worth landing early, and it is small.

_model_mismatch_error + the _meta provenance keys (lightrag_embedding_model, lightrag_embedding_dim), written into the mapping at create time alongside the existing ownership identity. This is the same-dimension-swap gate this issue asks for, already written for one of them.

"Absent evidence never refuses." An index recording no dimension or no model predates the provenance; silence must not be read as a mismatch or every pre-upgrade index is refused. This is the rule that makes the one-time marker decision tractable — and it is why a marker backfill is needed at all, since without one the silence never ends.

put_mapping replaces _meta wholesale, so every marker write must merge. Already true of _claim_index_for_workspace on main, and the PR re-derives it for its own marker write. The sharp corner it found: the operator repair command printed in the error message must carry the complete _meta too — a copy-pasteable PUT .../_mapping holding only the new key would strip the ownership identity and hand the index to any folding-equivalent deployment. If PR 2 backfills a model marker onto existing indices, both halves apply.

_consumption_marker_state's three-state answer — True / False / None. A marker write whose response was lost, and a response carrying acknowledged: false, are ambiguous, not negative: the cluster may have taken the change. Both must be re-read before being acted on, and "the re-read could not say" is a third outcome that must not collapse into either. Any backfill in PR 2 inherits this exactly. (27c698b04, 7daaa4f50.)

_declared_model_name() — trivial, but it is the single place that decides what "this instance's model" means (strip(), non-empty, str), and the value recorded must be the unfolded name. (Landed in PR 1 as lightrag/kg/vector_space.py::declared_model_name.)

Tests to carry over

These pin behaviour that survives, and their fakes are the reusable part — the PR drives initialize() against a small booking fake rather than a flat stub, so a check that runs against the newly created index sees the index it just created:

• test_attaching_refuses_an_index_built_by_another_model
• test_losing_the_create_race_still_validates_compatibility
• test_an_unreadable_mapping_after_the_create_race_fails_closed
• test_presence_recheck_refuses_a_foreign_model
• the _capture_lightrag_logs helper — the lightrag logger sets propagate=False, so caplog catches nothing without it

Findings for other PRs

• docs/ProgramingWithCore.md is wrong today, independent of this issue: it says switching embedding models requires clearing the data directory, which has not been true for Milvus / Qdrant / PostgreSQL since model isolation landed. The PR's split is not the right text under this issue's ruling (the answer is now "it fails closed, run lightrag-rebuild-vdb"), but the section does need rewriting — fold it into whichever PR lands the gate first.
• _assert_index_name_within_limit (OpenSearch refuses an index name over 255 bytes) was introduced because the suffix made the limit reachable. {workspace}_{namespace} can already overflow it with a long WORKSPACE, and main lets the cluster reject that with an opaque message. Optional, small, unrelated to the gate.
• The PR reached for the existing DataMigrationError for conditions that are not migrations. A signal for PR 1's naming: the typed vector-space-mismatch exception is a distinct condition and should not be folded into that type. (Done — VectorSpaceMismatchError.)

Explicitly NOT salvageable

The suffix itself; _plan_legacy_migration / _legacy_migration_source / _reindex_legacy / _mark_legacy_consumed; OPENSEARCH_MIGRATE_UNMARKED_LEGACY; the whole TestVectorLegacyMigration class; and the drop() docstring about sibling models' indices. All of it exists to spare an upgrade one lightrag-rebuild-vdb run, which is the tool this issue makes work.

One piece is worth remembering only if a reindex is ever revived: the copy used op_type=create with conflicts: proceed and counted completion as created + version_conflicts, so a peer's concurrent copy or already-started ingestion could not be overwritten and could not look like a short copy either.

Generate