`repair-status` reports `unknown` for healthy palaces with fewer records than `sync_threshold`

Summary

repair-status determines health by comparing the SQLite row count against index_metadata.pickle. ChromaDB only writes that pickle on persist, and persist is gated by hnsw:sync_threshold. So any palace holding fewer records than its sync_threshold has no pickle, and reports status: unknown forever — even though it is fully healthy and serving correct semantic results from the WAL.

With the 3.9.0 default of sync_threshold: 1000, this is every palace under ~1,000 drawers. For collections carrying a larger threshold from an earlier release's HNSW tuning, it persists much further up.

Observed on a 42,768-drawer palace (mempalace 3.9.0, chromadb 1.5.9), whose collection carries hnsw:sync_threshold = 50000 from an older release:

[drawers]
  sqlite count:   42,768
  hnsw count:     (no flushed metadata yet)
  status:         UNKNOWN
  note:           HNSW capacity unavailable: metadata has not been flushed;
                  leaving vector search enabled

hnsw_capacity_status() returns hnsw_count=None, divergence=None, status="unknown". Meanwhile the collection answers semantic queries correctly across all 42,768 drawers — verified with probes whose matches share no distinctive terms with the query, so lexical matching cannot explain the hits.

Why this matters

unknown is indistinguishable from unknown. A health check that cannot report OK for an ordinary small palace trains operators to ignore it — and it is the one command positioned to catch real trouble.

We hit this while investigating a palace that had genuinely degraded: WAL coverage had fallen to 8,223 of 42,730 live drawers (19%), so roughly four-fifths of the corpus was missing from search results. No error surfaced at any level for about 3.5 months. repair-status was the natural place to notice and it said unknown, exactly as it does when everything is fine.

Context on how that palace got there (both causes already fixed in 3.9.0 — noted only to show why detection mattered):

• _segment_appears_healthy() in 3.3.5 used data_level0.bin size as a record-count proxy. hnswlib preallocates that file to capacity, so every never-persisted segment was condemned. Fixed in 3.9.0 by the link_lists.bin discriminator.
• quarantine_invalid_hnsw_metadata() treated dimensionality=None as invalid. On chromadb 1.5.9 that is the normal persisted shape — we confirmed with a freshly built control collection whose pickle is identical in shape. Fixed in 3.9.0 by _missing_dimensionality_appears_recoverable() (#1710).

Independently each is survivable. Together, the second quarantined healthy populated indexes and the first blocked every replacement from surviving — 208 quarantines of one segment, 301 .drift-* directories. We have not proven the causal path from that churn to the WAL erosion, but the erosion is measured fact.

Suggested fix

Consult embeddings_queue when the pickle is absent. The WAL is the authoritative record of what is searchable — chroma replays it into the in-memory index at load — and it is readable with the same no-client, SQLite-only approach hnsw_capacity_status() already uses:

SELECT count(*) FROM embeddings_queue WHERE vector IS NOT NULL;

Comparing that against the live drawer count distinguishes the three states the current check collapses into one:

WAL coverage	pickle	true state	reported today
~100%	absent	healthy, sub-threshold	unknown
~100%	present	healthy, flushed	ok
19%	absent	degraded, losing recall	unknown

Rows 1 and 3 are the ones that matter, and today they are indistinguishable.

A stronger addition would be an optional end-to-end probe: a known-answer query asserting an expected drawer returns within a distance threshold. One caveat from our own debugging — such a probe must be lexically disjoint from its target, or hybrid search satisfies it via BM25 and the check passes while vector search is broken.

Not a useful signal: hnswlib's cur_element_count in header.bin. It reads 0 on a healthy sub-threshold segment, because it counts vectors flushed to disk rather than vectors that can be searched. It cost us an hour of misdiagnosis.

Secondary: quarantined segments are never reclaimed

quarantine_stale_hnsw() and quarantine_invalid_hnsw_metadata() rename segments to .drift-<ts> / .corrupt-<ts>, but nothing ever removes them. The palace above accumulated 307 such directories. Retaining them is right — they preserved the evidence that made this diagnosis possible — but a retention policy, or a mempalace clean surfacing total reclaimable size, would help. Worth considering alongside the fixes above, since a quarantine loop makes the growth unbounded.

Environment

• mempalace 3.9.0 (issue also present in 3.3.5)
• chromadb 1.5.9
• Python 3.12, Linux
• ~42.7k drawers, 384-dim, all-MiniLM-L6-v2