exclude: patterns in index.yml to prevent double-indexing nested collections
Nested collections double-index files and waste top-K slots. Need an exclude: key in index.yml.
Repro
collections:
wiki:
path: /Users/me/projects/MyRepo/wiki
pattern: "**/*.md"
specs:
path: /Users/me/projects/MyRepo
pattern: "**/*.md"qmd query "some topic" -n 10 --no-rerank --json returns:
qmd://wiki/syntheses/foo.md
qmd://specs/wiki/syntheses/foo.md # same file, different URIOn a real corpus (1190 docs, 195 in nested wiki), I just measured 2 of 10 top hits as literal file duplicates on a representative query. That's 20% of the visible top-K gone. Long-tail hits at ranks 7-10 get pre-empted by shadow copies.
Why nesting is reasonable
Not a config mistake. A wiki/ subdirectory of curated synthesis pages inside a larger repo of raw specs is a normal knowledge base shape. The two collections do different jobs (synthesized vs. primary) and need different context: strings, but they share a filesystem subtree. Asking users to physically separate the directories breaks the workflow.
Proposal
collections:
specs:
path: /Users/me/projects/MyRepo
pattern: "**/*.md"
exclude:
- "wiki/**"
- "node_modules/**"
- ".git/**"Standard glob, applied after pattern, evaluated relative to the collection's path. Optional --exclude flag on qmd collection add would be nice but isn't required, editing index.yml works.
Workaround today
Consumers can dedupe by stripping the qmd://<collection>/ prefix and suffix-matching path tails post-retrieval. I'm doing this in my downstream skill now. Two problems with it:
- Every consumer reinvents the same logic.
- Duplicates still occupy candidate slots before rerank, so the deep path pays compute cost on docs that get thrown away.
Risk if wrong
If exclude: semantics differ from pattern: semantics in subtle ways (relative-to-path vs. absolute, glob dialect mismatch), users will hit it. Worth being explicit in docs that exclude is evaluated against paths relative to the collection's path, same dialect as pattern.
Version
qmd 2.1.0 (6f293daa9f)
Source: tobi/qmd