The new relevance floor does not reject coherent off-topic queries, and fails entirely on general-prose corpora

Summary

v1.35.0-rc.1 added a retrieval-decline mechanism. It reliably rejects noise, but it does not reject coherent off-topic questions, and on a library containing a large general-prose corpus it does not fire at all.

Filing this as measurement data rather than a proposed fix, since the retrieval path is under active work.

What shipped

RAG_DEFAULT_SCORE_THRESHOLD = 0.3 (the Qdrant recall cutoff) is unchanged. The new mechanism is a second, post-rerank floor:

• RAG_MIN_FINAL_SCORE = 0.62 in admin/constants/ollama.ts
• applied by applyRelevanceFloor() in admin/app/utils/misc.ts, called from rag_service.ts
• tunable via KV rag.minRelevance (0 off / 0.55 lenient / 0.62 balanced / 0.66 strict)

Measured results

Three servers on v1.35.0-rc.1, differing only in installed content and chat model. Same four prompts on each. Cells show the citations returned.

Query	Box A — devdocs only (small, technical)	Box B — Project Gutenberg + news	Box C — gardening StackExchange + devdocs (117k vectors)
zxqw plorbnat frunkle wibbet 7734 quozz	declined	Wikivijesti; Gutenberg	declined
Reply with exactly the single word: banana	Node.js Docs	Gutenberg; Node.js Docs	JavaScript Docs
What is the capital city of France?	declined	Gutenberg ×2	JavaScript Docs
My sister's birthday is next Tuesday. What should I get her?	declined	Gutenberg	CSS Docs; Python Docs

Models: llama3.1:8b on A and B, qwen2.5vl:3b on C — so this is not model-specific.

The part that may not be obvious

Effectiveness scales inversely with how much content is installed.

Box A's library is small and narrowly technical, so little clears the floor. Box B holds Project Gutenberg — a large corpus of general English prose in which something is semantically near any input — so the floor never fires, and even gibberish is cited.

That means the better-stocked the appliance, the worse the gate performs, and the flagship collections (Wikipedia, Gutenberg) are the worst case. Testing on a lightly loaded box will understate this.

Secondary observation

applySourceDiversity runs after the floor, so its 0.85^n penalty can push a survivor back below 0.62 without it being dropped.

Files involved

• admin/constants/ollama.ts — RAG_MIN_FINAL_SCORE, RAG_DEFAULT_SCORE_THRESHOLD
• admin/app/utils/misc.ts — applyRelevanceFloor
• admin/app/services/rag_service.ts — floor application and applySourceDiversity ordering