Query-side LUT is quantized to 7 bits (max_lut=127), costing 1.5-3.5pp recall@10 — a cross-arch equivalence trade, not a rounding detail

Found by bug-hunt wave 22 (numerical-stability lens). Severity: medium-high — this is the dominant error term in the whole scoring path, 2-4 orders of magnitude larger than every float-arithmetic effect, and it's undocumented as an accuracy trade.

Method (so the numbers are checkable): an independent NumPy reference reimplementing ChaCha8 + fisher_yates + the 2-round permute/sign/WHT rotation, parsing the v6 .tv bytes directly (codebook, blocked codes, per-vector scales, TQ+ arrays), then computing three scores per (query, vector): S_exact (f64 exact decode), S_lut (bit-faithful u8 LUT per search.rs:1093-1187, summed in exact integer + f64), and S_tv (what search() returns). S_lut reproduces S_tv to ≤3e-6 relative everywhere, so the simulation is faithful.

turbovec/src/search.rs:1170 — let max_lut: f32 = 127.0; with a single global scale = max_span / max_lut.

|S_lut − S_exact| relative to the median top-k score (purely query-side — same codes):

dim	2-bit	3-bit	4-bit
128	3.0 %	4.3 %	5.5 %
768	4.0 %	5.2 %	5.7 %
3072	5.5 %	6.0 %	7.5 %
8192	4.6 %	7.2 %	10.4 %
16384	5.4 %	8.2 %	10.0 %

Practical impact: the returned top-10 differs from an exact f64 decode of the very same codes in 70-100% of queries (set overlap 0.93-0.98). Against true float ground truth at 4 bits: recall@10 0.845 vs 0.870 (dim 768), 0.845 vs 0.860 (3072), 0.855 vs 0.890 (16384) — 1.5-3.5pp of recall discarded by the LUT alone. At 2 bits it's in the noise (code quantization dominates).

The comment at search.rs:1157-1169 says x86 could safely carry max_lut ≈ 255 and that 127 is chosen only so ARM- and x86-built indexes round identically. That cross-arch equivalence is costing ~1 bit of query precision and ~3pp of recall — worth surfacing as a deliberate trade rather than a rounding detail. (The loss isn't ARM's vaddq_u8 cap per se; it's x86 being held down to match.)

Second finding — a single global max_span makes the LUT collapse for concentrated rotated queries (search.rs:1172; mechanism CONFIRMED, real-world reachability low). One scalar is shared by all dim sub-tables, so one wide sub-table starves the rest. Constructed queries whose rotated form is concentrated (unit_query + a·R⁻¹e_j; ROTATION_SEED is a public constant at rotation.rs:82), dim 768, 4-bit:

spike a	rot. concentration	span_max/median	eps_lut / score	top-10 overlap
0	0.12	5	0.057	0.97
1	0.70	41	0.46	0.72
3	0.95	124	2.6	0.67
10	0.995	414	3.5	0.68

At the worst point the LUT error is 3.5× the score magnitude and a third of the top-10 is wrong. Natural queries rotate to concentration ~0.12 and never approach this, so exposure is low — but it fails silently with no warning or fallback, and the rotation is a fixed public constant so it's reachable by construction. A per-sub-table scale, or detecting span_max/median > ~50, would guard it.

Everything else in the float pipeline is sound (measured bounds, recorded so nobody re-derives them): kernel f32 accumulation |S_tv − S_lut| maxes at 2.6e-4 relative at dim 16384 with 0/180 queries reordered, and only 1/10 in a deliberately adversarial near-tie set; the Walsh-Hadamard is flat at ~1.5e-7 relative at every block size including weak-block dims, with no compounding from the 2-round structure; the calibration bias is computed in f64 and the cancellation regime is self-limiting (bias/score falls from 0.76 to 3.4e-5 as the data offset grows, since TQ+ absorbs the offset into shift); and f32 scale storage has 208× overflow headroom at MAX_DIM (worst case 1.64e36 vs f32::MAX 3.4e38 — correctly placed, though the margin in value is only ~14×, not the ~1e22 one might assume).

Readability trap worth a comment: the lo/hi sub-table naming in build_query_neon_lut_from_slice (search.rs:1114-1140) is inverted relative to the packing in build_extract_lut (pack.rs:225-243) — dims [g·cpb, g·cpb+cpn) land in the high nibble but are served by the table labelled lo. Consistent end-to-end, so purely cosmetic, but it cost this agent a debugging cycle and #314 flagged the same inversion independently.

Generated with Claude Code