[BM25] Tracking issue: BM25 scoring on the text payload index

Tracks scoring BM25 directly off the full-text payload index, instead of embedding text into a sparse vector.

The index already holds the tokens, df and N; this work adds:

• document lengths
• corpus statistics gathered across segments
• MaxScore scoring

avgdl comes from the data instead of a client-supplied constant, and existing payload fields become rankable without re-importing anything.

Scoring is opt-in per text index through the scoring param (#10770), which records the lengths. The Query API reaches it from #10786: {"query": {"text": "..."}, "using": "<field>"}.

Related: #9779, #10285, #10286

---

Merged:

• #10614
• #10616
• #10639
• #10675
• #10677
• #10670
• #10673
• #10798
• #10688
• #10760

---

Open (stacked, bottom up, in merge order):

• #10810
• #10811
• #10689
• #10761
• #10762
• #10764
• #10765
• #10767
• #10770
• #10773
• #10782
• #10784
• #10785
• #10786

---

Not started:

• BM25 as a rescore of prefetched points (refused by the planner today)
• BM25 on edge (refused today)

Before release:

• User docs: the scoring index param, the text query (OR semantics, k / b), and a sparse BM25 to text index migration note

Done on the stack:

• The on-disk token total is stored in the postings header at build time (#10798, replaces #10759), so the statistics gather no longer reads the doc-length sidecar per query; deletions since the build stay in it, as they stay in df, until the next optimization
• A postings header written before it carried the total reports no total, rather than a zero that would skew avgdl (#10798)
• The per-segment scorer is split into the core over the mutable index (#10810), the immutable and on-disk shapes (#10811) and the FullTextIndexRead wiring (#10689); the stack is linked in that order on GitHub
• Append-only deletion never reaches the field index: N and total_tokens kept counting tombstoned points (#10782)
• Deferred and shadowed points no longer count in N and total_tokens (#10784); a proxy segment's deleted points still do during an optimization, accepted as for sparse IDF
• Shard-level text benchmark next to the sparse baseline (#10785): at 200k, text is 2.3x / 1.6x / 1.5x faster (fresh / optimized / on-disk) at recall 0.994 vs 0.718
• Query interface (#10786): a canonical TextQuery ({"text": "..."} or {"text": {"query", "k", "b"}}), using resolved against payload fields only, OR semantics
• A text query no longer drops a corpus-scoped idf silently: refused at the root and as a prefetch (#10767)

Performance follow-ups:

• Parallelize the segment loop of fill_query_context, for both the text statistics and the sparse IDF gather
• Batch the per-term posting_len reads of the statistics gather (one round trip per term per segment on a remote disk)
• Measure block length reads on a backend with real per-read latency, and pick ON_DISK_BLOCK from it
• Batch the scorer's filter checks through check_batched per block (one payload read per candidate on disk today)
• Quantize document lengths to one byte (Lucene-style norm): 4x smaller length sidecar, fewer bytes per batched read; a format change once scoring indexes exist

Test follow-ups:

• Model tester: it runs with FeatureFlags::default(), so it never exercises append-only deletion or deferred points; #10782 and #10784 are pinned by segment tests only

Test follow-ups done in #10770 (possible once params carry scoring):

• Direct tests for the new_mmap and open_immutable scoring guards (a missing or rejected length sidecar under scoring) (#10770)
• test_congruence: assert doc_len_batch and total_tokens after the reopen too, not only before (#10770)
• Mark the four positions off, lengths on tests as TF-only-tier guards, since scoring now implies positions (#10770)

Engine findings outside BM25:

• The indexing optimizer decides from vector storage alone, so a shard holding only payloads never leaves its appendable segment and its text index stays mutable (found by #10785)

Not in scope yet:

• corpus-scoped statistics (every point of the shard for now), see #9445
• cross-shard IDF
• BM25F, prefetch boost, score_fusion norm
• retiring the sparse BM25 route