recall.scoreThreshold applied across incommensurable score scales (BM25/cosine 0-1 vs RRF ~0.03)

Summary

recall.scoreThreshold (default 0.3) is applied as a single absolute cutoff, but the scores it is compared against live on different, incommensurable scales depending on the search strategy.

Details

• keyword (FTS/BM25 via bm25RankToScore): 0–1 scale → 0.3 is a plausible cutoff.
• embedding (cosine, 1 - distance): 0–1 scale → 0.3 is plausible.
• hybrid (RRF, k=60): scores are ~0.016–0.05 (RRF of two lists). An absolute 0.3 cutoff here would filter out essentially everything.

So the same scoreThreshold constant means very different things per strategy. In practice the hybrid path avoids the problem only because it does not apply the threshold at all (see related issue about searchHybrid ignoring _threshold). The keyword path additionally has a "document set is small → return below-threshold results anyway" fallback, which masks the scale mismatch.

Suggestion

Pick one of:

1. Apply per-source thresholds before fusion (cosine ≥ x, BM25-score ≥ y) instead of one absolute value on the fused score;
2. Drop the absolute threshold for fused/RRF results and rely on top-N;
3. At minimum, document the per-strategy meaning of scoreThreshold so operators do not set a value that is a no-op or over-filters depending on strategy.