Query sampling framework for kNN search

Right now, we have no way to estimate what recall (or other metrics) a user’s query traffic would achieve in production kNN searches. Offline benchmarks use query sets that look nothing like live traffic: live traffic is heavily skewed (Zipfian), full of repeats, and has sub-populations (filtered queries, hard queries in dense regions) whose recall can be far below the traffic-weighted average. In the reference run filtered queries were at Recall@10 0.2–0.3 while the traffic-weighted number was 0.91.

The idea is to keep a small, continuously maintained sample of live kNN queries with (almost) exact ground truth, plus the design weights needed to turn it back into unbiased estimates for the live population. That sample is then the input for recall monitoring, knob tuning and regression checks.

How this differs from the query log

We already have a coordinator-side query log (elasticsearch.querylog.*, ActivityLogger wrapping TransportSearchAction). It is not a good base for this because:

• it logs every query above a took-time threshold, it does not sample;
• it serialises source.toString() synchronously on the response thread;
• it keeps no top-k ids/scores and no parsed kNN parameters.

The slow log is shard-level and never sees the final response, and search usage stats are counters only. So the capture point is a new ActionFilter on indices:data/read/search, but it follows the same conventions as the query log (dynamic cluster settings, skip requests coming from a remote cluster).

Pipeline

1. Capture gate: Bernoulli gate at rate ρ with a per-hour floor; non-blocking hand-off; drops are counted, never blocking.
2. Dedup + multiplicity: 128-bit fingerprint of (vector, canonical filter), windowed w(q), inclusion-probability accumulator.
3. Stratified sampler: log damping on w(q), inverse cluster population (α), standardised-LID tilt (β), head rule, acceptance token bucket with continuous γ feedback.
4. Tier 1 / Tier 2: short TTL buffer + uniform event slice; exact ground truth off the search thread under a cost budget; golden dataset with versioning.

Plan

• 1 Plugin skeleton: settings and per-node stats API
• 2 Zipfian traffic generator (external, black box to the framework) — lives outside the repo
• 3 Stage 1: capture filter, Bernoulli gate with floor, non-blocking hand-off
• 4 Stage 1: JMH benchmark for gate overhead
• 5 Stage 2: fingerprinting, counting TTL map, multiplicity and inclusion probability
• 6 Stage 3a: stratification (damping, spatial, hardness, head rule)
• 7 Stage 3b: acceptance token bucket and γ feedback
• 8 Tier 1 buffer and uniform event slice
• 9 Ground-truth worker with cost credit and sampling_cost_ratio
• 10 Tier 2 golden dataset
• 11 Estimators API (Horvitz–Thompson recall, Kish ESS, observed production recall)