skill-creator: parallel trigger-eval workers share one commands dir, collapsing measured trigger rate to ~1/num_workers

Summary

skill-creator's trigger-eval harness writes every worker's temp command file into a single shared directory, so parallel workers see each other's files. Each claude -p process is presented with N identical skills differing only by UUID, picks one, and is scored as "triggered" only if it happens to pick its own.

The measured trigger rate therefore collapses toward 1/num_workers independently of the description being tested. Since --num-workers defaults to 10, run_loop.py optimizes against roughly 10%-scaled noise.

Affected code

skills/skill-creator/scripts/run_eval.py, run_single_query():

project_commands_dir = Path(project_root) / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"

project_root comes from find_project_root(), which walks up from cwd. It is the same path for every worker, and workers run concurrently via ProcessPoolExecutor. Detection then requires the run's own clean_name:

if clean_name in accumulated_json:
    return True

A run where the model triggers correctly but selects a sibling worker's identical skill is recorded as a non-trigger.

Evidence

Same eval set (20 queries), same skill, same description, same model, varying only --num-workers:

Recall tracks 1/N. The description was never the variable.

Two independent confirmations:

1. An isolated manual probe — one command file, one claude -p, no concurrency — triggered on the first attempt for the query "boot the whole ineedabaker stack…". Replaying run_eval.py's own stream-parsing logic over that transcript returns True. The harness scored that identical query 0/3 under default settings, twice.
2. During a --num-workers 3 run, the host surfaced two simultaneously registered skills differing only by UUID (run-ineedabaker-skill-5d2f8649, run-ineedabaker-skill-e9edc6a8). The directory is empty after each run, so cleanup works — the collision is purely concurrent.

Impact

run_loop.py selects best_description by comparing scores that are dominated by this race. In our run it preferred a rewrite that was not better, and reported 6% recall for a description that measures 80% when evaluated correctly. Any conclusion drawn from the default configuration is unreliable.

Reproduction

cd skills/skill-creator
python -m scripts.run_eval --eval-set evals.json --skill-path <skill> \
  --num-workers 10 --runs-per-query 1 --verbose   # recall collapses
python -m scripts.run_eval --eval-set evals.json --skill-path <skill> \
  --num-workers 1  --runs-per-query 1 --verbose   # recall recovers

Suggested fix

Isolate each worker rather than sharing one commands directory — give every invocation its own temporary project root (tempfile.mkdtemp() containing .claude/commands/) and pass it as cwd to the subprocess. That keeps parallelism while guaranteeing exactly one candidate skill is visible per run.

Serializing (--num-workers 1) also works but makes the 5-iteration loop far slower than the docs imply.

Worth noting separately: --timeout defaults to 30s while 10 workers contend, which independently truncates slower runs. Isolation fixes the race; the timeout default may deserve its own look.

Environment

• anthropics/skills @ 3b3fad96af16a10759d930941b4520ba0c40edae
• Claude Code 2.1.251, macOS 15 (Darwin 25.6.0), Python 3.13
• Model: claude-opus-5