skill-creator: trigger detection reports 0% recall for every skill

Summary

skill-creator's trigger evaluation reports precision=100% recall=0% for every skill, regardless of how good the description is. run_loop then "improves" the description against evidence that every positive query failed, and reports the result as an optimisation.

This fails silently. Nothing errors, and the output looks like a legitimate measurement of a bad description.

Cause

run_single_query() in skills/skill-creator/scripts/run_eval.py decides whether the skill triggered by watching the claude -p event stream. Three paths return "did not trigger" before the evidence can arrive.

1. Any other tool appearing first is treated as proof of failure (line ~137):

if tool_name in ("Skill", "Read"):
    pending_tool_name = tool_name
    accumulated_json = ""
else:
    return False

A realistic query makes the model look around first — Bash, Glob, Grep. The harness sees that first tool, returns False, and kills the process. This bites hardest on exactly the queries the skill-creator guide asks authors to write: concrete ones, with file paths and backstory.

2. The first content_block_stop ends the whole judgement:

elif se_type in ("content_block_stop", "message_stop"):
    if pending_tool_name:
        return clean_name in accumulated_json

A Read of an unrelated file before the Skill call closes its own block, and the match is decided against that block's accumulated input. A later Skill call is never seen.

3. The non-streaming fallback returns inside its own loop:

for content_item in message.get("content", []):
    ...
    return triggered

The return is inside the for, so only the first tool_use in an assistant message is examined.

Evidence

Found while tuning the description of reveal-doc-style, an internal house-style skill for protocol and regulatory prose at Reveal Genomics — it carries the writing rules and the tracked-changes tooling used on a diagnostic assay's bridge study plan.

Raw stream for a query the harness scored as not triggering:

query:             "i went through the bridge study plan last night and made a bunch of edits in
                    word. can you work out what i changed and roll it into section 7"
tool_use order:    ['Skill', 'Bash', 'Bash', 'Bash', 'Read']
input_json_delta:  '{"skill": "reveal-doc-style-skill-test01'

That query is representative of the whole set: concrete, with a real document and a real section number, which is exactly what makes the model orient itself with Bash before or around the Skill call.

The skill was the first thing the model reached for, and the harness still scored it 0.

Measured on a 20-query trigger set for reveal-doc-style — 11 should-trigger, 9 deliberately near-miss negatives (a warehouse query, a docx-to-markdown conversion, a slide deck, a literature lookup) — serially, one run per query:

	shipped	patched
positives detected	0 / 11	5 / 11 (original description)
positives detected	—	9 / 11 (revised description)
false alarms	0 / 9	0 / 9

The patched harness is what showed the original description was under-triggering, and what confirmed a revision fixed it. The shipped harness reported both as identically worthless.

Fix

Keep scanning until the result event: don't treat other tools as disproof, reset per-block state instead of concluding from it, and return from the fallback only after the whole message has been walked. Happy to send a PR if useful.

Also, possibly separate

With --num-workers above 1 the patched harness still returned zeros on queries that pass serially. Not diagnosed; all numbers above are --num-workers 1. Worth checking whether concurrent claude -p processes sharing one .claude/commands/ directory interfere.

Environment

Claude Code on Windows, running the harness under WSL. claude plugin CLI available; anthropic-skills plugin.