[Bug]: Live records become unreachable by query() after deletes and upserts while get() still returns them (1.5.9)
What happened?
After a collection goes through many deletes and upserts, a few live records are never returned by query(), even when the query embedding is the record's own stored embedding and n_results=1000. get(ids=[...]) still returns each of them. Reopening the store in a new process does not make them reachable again.
This looks like the same symptom as #3580 (reported there on 0.4.14 at ~21M records, without a reproduction). Below is a small, self-contained reproduction on 1.5.9.
Environment: chromadb 1.5.9, PersistentClient, default HNSW settings, hnsw:space=l2, Python 3.12.7, Ubuntu 24.04.
Reproduction (~1 min): 4,000 inserts → delete 50% → upsert 30% of the remaining records → for each live record, query with its stored embedding (top-100 and top-1,000) → reopen the store in a fresh Python process and repeat.
repro_chroma_unreachable.py"""Standalone reproduction for the disclosure: live records unreachable by search
in Chroma after heavy deletes/upserts. No vfo dependency. ~1 minute.
pip install chromadb numpy && python repro_chroma_unreachable.py [seed]
Builds the collection, lists live records that a query with their own stored vector
does not return (top-100 and top-1,000), then reopens the store in a fresh Python
process and checks again. The affected records differ between runs even with the
same seed (index construction is not deterministic), so run it a few times.
"""
import subprocess, sys, tempfile
import chromadb, numpy as np
def unreachable(col, k_top):
g = col.get(include=["embeddings"])
out = []
for i, e in zip(g["ids"], g["embeddings"]):
r = col.query(query_embeddings=[list(e)], n_results=k_top)
if i not in r["ids"][0]:
out.append(int(i))
return len(g["ids"]), sorted(out)
def check(path, label):
col = chromadb.PersistentClient(path=path).get_collection("repro")
n_live, miss100 = unreachable(col, 100)
_, miss1000 = unreachable(col, 1000)
print(f"{label}: {n_live} live records; not returned for their own vector: "
f"top-100 {len(miss100)} {miss100}, top-1000 {len(miss1000)} {miss1000}", flush=True)
for k in miss1000:
g = col.get(ids=[str(k)], include=["embeddings"])
print(f" id {k}: get() found={bool(g['ids'])}", flush=True)
if len(sys.argv) > 2 and sys.argv[1] == "--check":
check(sys.argv[2], "after reopen in a new process")
sys.exit(0)
seed = int(sys.argv[1]) if len(sys.argv) > 1 else 3
rng = np.random.default_rng(seed)
path = tempfile.mkdtemp()
col = chromadb.PersistentClient(path=path).get_or_create_collection("repro", metadata={"hnsw:space": "l2"})
n, d = 4000, 32
X = rng.standard_normal((n, d)).astype(np.float32)
for i in range(0, n, 500):
col.add(ids=[str(j) for j in range(i, i + 500)], embeddings=X[i:i + 500].tolist())
live = set(range(n))
dele = rng.choice(n, n // 2, replace=False)
col.delete(ids=[str(int(k)) for k in dele]); live -= set(dele.tolist())
up = rng.choice(sorted(live), int(0.3 * len(live)), replace=False)
col.upsert(ids=[str(int(k)) for k in up], embeddings=rng.standard_normal((len(up), d)).astype(np.float32).tolist())
print(f"chromadb {chromadb.__version__}, seed {seed}", flush=True)
check(path, "same process")
subprocess.run([sys.executable, __file__, "--check", path], check=True)Observed (five runs with seed 3; ids of live records absent from the top-1,000 for their own embedding; get() found every one):
| run | same process | after reopen in a new process |
|---|---|---|
| 1 | 7: 446, 522, 1051, 1442, 1538, 1903, 2069 | not measured |
| 2 | 5: 446, 784, 1051, 1442, 1505 | not measured |
| 3 | 4: 446, 683, 1442, 1538 | 5: 292, 446, 1442, 1538, 2767 |
| 4 | 5: 446, 1442, 1538, 1903, 3911 | 5: 446, 1442, 1538, 1903, 2475 |
| 5 | 8: 143, 292, 446, 1051, 1442, 1538, 1971, 2908 | 8: 1, 446, 1051, 1442, 1538, 2069, 2475, 2908 |
The affected ids vary between runs with the same seed, but 446 and 1442 were affected in every run. A longer randomized insert/upsert/delete workload with crash-restarts showed the same pattern: some flagged records became searchable again after a restart, others did not.
Expected behavior
Every live record is returned by a query with its own embedding (distance 0), at least with a large n_results; or the documentation states that records can become unsearchable after deletes and that a rebuild restores them.
Likely mechanism
Deleted nodes stay in the hnswlib graph (allow_replace_deleted is false, see #2594), and nodes whose neighbourhoods are mostly deleted become disconnected from the entry point (the "unreachable points" problem, arXiv:2407.07871).
Source: chroma-core/chroma