[FEATURE] Hybrid BM25 + cosine retrieval (RRF, current retrieve path)

Description

Local progressive_retrieve ranks L2 segments and workspace resources by cosine only. After the query is embedded, the original text is discarded. Exact tokens (hostnames, error codes, CJK terms, file paths) miss if the embedder does not rank them in top_k.

ADR 0007 already specified a single-pass hybrid (embedding + BM25) over L2 items, then a roll-up to L1. This issue is that slice on the current retrieve path — not the wiki-graph kernel.

Motivation

Inject (<binary> retrieve) runs this every turn. Keyword misses are user-visible: the store contains ECONNREFUSED 127.0.0.1:5432 and the query says 5432, but a semantically closer “db timeout” fragment occupies the five cosine slots.

CONTRIBUTING lists retrieval performance as High. Segment text and resource caption are already stored; BM25 needs no new column and no LLM.

Platform

Platform Independent (local inmemory / sqlite / postgres). Cloud retrieve is unchanged (server-side).

Priority

Major

Proposed design

Happy to change any of these if maintainers prefer an alternative — say so on the issue or PR.

1. Candidate set. sqlite/inmemory already scan the scoped corpus → score BM25 over that whole pool. Postgres segments keep the pgvector index: take cosine candidates ∪ BM25 candidates (candidate_k = max(50, 5 * top_k)), RRF, cut to top_k. We will not fuse only inside cosine top_k (that cannot surface keyword hits the embedder ranked out). Postgres resources today already brute-force in Python, so they use the full-pool path until they get pgvector.

2. Fusion = RRF (1/(60+rank)), not ADR 0007’s min-max linear blend. The ADR never specified α; min-max on k=5 is unstable. RRF needs no α. Cosine-only remains via MEMU_RETRIEVE_HYBRID=0.

   Also considered: min-max + α=0.7.

3. On by default. That is the product point. Kill switch: ProgressiveRetrieveConfig.hybrid=False / MEMU_RETRIEVE_HYBRID=0.

   Also considered: default off, opt-in.

4. Segments and resources in v1. File roll-up stays max(segment score), so files benefit for free. Resource captions (paths, package names) are at least as keyword-heavy as fragments.

5. CJK n-grams from the start (unigrams + bigrams on CJK runs; lowercase latin/digit tokens). No jieba.

   Also considered: English/punctuation only.

6. Repo ranks, not the service. vector_search_segments / vector_search_resources take optional query_text so each backend can pick full-pool vs candidate-union. Fusing in agentic.py would force Postgres to ship the whole corpus.

ADR 0007’s kernel is untouched. A new ADR records that current retrieve uses RRF so the docs match the code (existing ADRs are historical; we will not rewrite 0007).

Out of scope

• Wiki-graph kernel / L1–L2 redefinition (ADR 0007 remainder)
• Temporal decay (#630 — status-now, not a timeline mix-in)
• Postgres FTS / tsvector
• Changing file-layer search (still a roll-up)
• Cloud progressive_retrieve