bench: pathsMatch is an unanchored suffix test — a fixture naming a file that does not exist scores f1 1.0

Summary

qmd bench credits a query as correctly answered when the retrieved document merely ends with the expected path — in either direction. A fixture can name a file that does not exist anywhere in the corpus and still score f1 = 1.0, with the phantom path listed under matched_files.

This is the same unanchored-suffix hazard reported for the multi-get comma-list resolvers in #759 (fixed); it survives in the benchmark scorer, where it is arguably worse, because the whole point of a benchmark number is that it is trustworthy.

Present in 2.8.3 (current published).

Reproduction — CLI only, no internals

Standards.md does not exist anywhere in the indexed vault (find . -name 'Standards.md' → 0 results). The real note is Memory/Procedural/Vault Note Standards.md.

{ "collection": "ai-live",
  "queries": [ { "id": "ghost", "query": "Vault Note Standards", "type": "exact",
      "expected_files": ["Standards.md"], "expected_in_top_k": 3 } ] }

$ qmd bench ghost.json --json
  f1        = 1
  matched_files            = ["Standards.md"]
  unmatched_expected_files = []

A perfect score for a document that does not exist.

Mechanism

dist/bench/score.js:24:

export function pathsMatch(result, expected) {
    const nr = normalizePath(result);
    const ne = normalizePath(expected);
    if (nr === ne) return true;
    if (nr.endsWith(ne) || ne.endsWith(nr)) return true;   // ← unanchored, both directions
    return false;
}

pathsMatch("notes/archive/Answer.md", "Answer.md")         // true
pathsMatch("Answer.md", "notes/archive/Answer.md")         // true — expected longer than result

scoreResults calls it for matched_files, recall, recall_at_k, hits_at_k and mrr, so a suffix collision inflates every metric, not just f1.

Why it bites in practice

Duplicate basenames across folders are ordinary in note vaults — common enough that linters check for them specifically. Two consequences, both silent:

1. A fixture is credited to the wrong document. The expected note can be absent from the results entirely while a same-basename sibling earns full marks. Comparing f1 alone will not reveal it: with expected Answer.md, a decoy Shadow/Answer.md at rank 4 and k=3, the scorer reports recall 1 and mrr 0.25 while f1 is 0 — so the corruption hides inside a metric that looks like an ordinary miss.
2. A typo in a fixture never surfaces. It resolves to whatever shares the suffix, so the fixture silently measures a different question from the one it names.

The second direction (ne.endsWith(nr)) is the more surprising half: an expected path longer than the returned one also matches, so a result of README.md satisfies an expectation of docs/api/README.md.

Suggested fix

Compare identity, not resemblance. After normalizePath strips the qmd://<collection>/ prefix, the remaining string is already collection-relative, so plain equality is the right test:

export function pathsMatch(result, expected) {
    return normalizePath(result) === normalizePath(expected);
}

If a lenient mode is wanted for hand-written fixtures, it would be safer as an explicit opt-in (--loose-paths) than as the default, and it should at minimum be anchored at a path separator so Standards.md cannot match Vault Note Standards.md.

Two related points, either of which would also have caught this:

• normalizePath lowercases. That makes scoring case-insensitive while the index is case-sensitive, so two documents differing only in case are indistinguishable to the scorer.
• Nothing warns when an expected_files entry matches no indexed path. Since a suffix match makes the phantom look satisfied, a fixture can rot indefinitely. A coverage check against the index's own path strings catches it — and it has to compare against the index rather than the filesystem, because existsSync on macOS is Unicode-normalisation-insensitive while the matcher is byte-exact.

Environment

• @tobilu/qmd 2.8.3 (npm), macOS 15, Node 25
• Corpus: 5 collections, 562 documents