README's recall claim holds for real embeddings but loses to FAISS on isotropic Gaussian and weak-block sparse

Found by bug-hunt wave 37 (consolidated recall measurement). Severity: medium — the headline claim is intact where it's stated, but it reads as universal and isn't.

Using the official suite methodology (n=100k, 1000 queries, seed 42, FAISS IndexPQ m=d/4 @2-bit and m=d/2 @4-bit, nbits=8, IP), with the methodology control passing (openai-1536 2-bit R@1 0.903 vs published 0.891; GloVe 2-bit 0.5695 vs 0.5637; GloVe 4-bit 0.8553 vs 0.8498).

The claimed configs reproduce — all wins or ties at R@1: openai-1536 2-bit 0.903 vs 0.867 (+3.6), 4-bit 0.970 vs 0.969; openai-3072 2-bit 0.931 vs 0.906 (+2.5), 4-bit 0.976 vs 0.971; GloVe 2-bit 0.5695 vs 0.5643, 4-bit 0.8553 vs 0.8410.

Configs where turbovec loses to FAISS IndexPQ (reproducible across two seeds):

config	turbovec	FAISS	delta
isotropic Gaussian d=768, 2-bit, R@1	0.353 / 0.393	0.450 / 0.490	−9.7 / −9.7 pp
isotropic Gaussian d=1536, 2-bit, R@1	0.440 / 0.363	0.450 / 0.453	−1.0 / −9.0 pp
isotropic Gaussian d=768/1536, 4-bit, R@1	0.757–0.783	0.803–0.817	−3.3 to −6.0 pp
sparse d=1032 (weak block), 2-bit, R10@10	0.546	0.757	−21.1 pp (turbovec still wins R@1)
GloVe d=200, 2-bit, R10@10	0.6245	0.6325	−0.8 pp

Important context that argues against over-reacting: on realistic anisotropic data turbovec crushes FAISS (d=768 2-bit R@1 0.697 vs 0.120), so the isotropic-Gaussian loss is a synthetic-regime effect, not a user-facing break. The README claim is correct for real embeddings. The gap is that it reads as universal, and "isotropic Gaussian" and "weak-block sparse" are the two counterexamples a skeptical reader would find. A sentence scoping it to real embedding distributions would close this. (Complements #310, which found the bimodal/one-hot losses.)

One methodological note worth recording for anyone re-measuring #332: I tried to isolate the 7-bit query LUT's contribution by injecting score noise at the measured 4–5.7% relative level. That costs 25–45 pp — far more than #332's measured 1.5–3.5 pp. Not a contradiction: it shows the real LUT error is strongly correlated across candidates (one LUT per query), which is exactly why its ranking damage is small. The emulation is invalid as a proxy; #332's number stands as measured there.

Also confirmed and sharpened: #335's magnitude cliff fires a decade earlier than reported — at query scale 1e-10, not 1e-11 (R10@10 0.926 → 0.0003 on the base config at 4-bit).

Recommended recall fix order (see the measurement table posted on #309): weak-block dims first (13–38 pp, up to 50 pp at R@1, and it's a free fix), then the magnitude cliff (#335 — cheap, catastrophic when hit), then the 7-bit LUT (#332 — 1.5–3.5 pp for every 4-bit user, the largest unconditional term after the irreducible quantization floor), and only then the calibration warm-up (#317/#366 — real but 1.5–1.9 pp at 2-bit on real data and ≤0.4 pp at 4-bit; worth fixing for its silence and irreversibility, not its magnitude).

Generated with Claude Code