skill-creator: run_eval.py parallel workers cross-match skill UUIDs, producing false-negative trigger rates

Summary

skills/skill-creator/scripts/run_eval.py produces systematically wrong (near-zero) trigger rates whenever --num-workers > 1 (the default is 10). The failure is silent: every query — including ones that obviously should trigger — comes back with trigger_rate ≈ 0, which then makes all should_trigger: false cases "pass" trivially and all should_trigger: true cases "fail". The aggregate looks like a bad skill description when in fact the measurement is broken.

Root cause

run_single_query writes each query's temp command file into a shared project commands dir and runs every claude -p in the same project:

clean_name = f"{skill_name}-skill-{unique_id}"          # line ~52  (per-query UUID)
project_commands_dir = Path(project_root) / ".claude" / "commands"   # line ~53  (SHARED)
command_file = project_commands_dir / f"{clean_name}.md"
...
cwd=project_root,                                        # line ~89  (SHARED)

Detection then requires the model's invoked skill name to contain this query's UUID:

if clean_name in accumulated_json:   # line ~147
    return True
...
return clean_name in accumulated_json   # line ~152

With ProcessPoolExecutor(max_workers=num_workers) (line ~198) running ~10 queries concurrently, up to ~10 command files — identical descriptions, different UUIDs — coexist in the shared .claude/commands/ dir. Each claude -p sees all of them and invokes one fleet of the skill, but the per-query check only matches its own UUID. The model almost always triggers a sibling worker's UUID → the query records a false negative. Result: trigger rate collapses to ~0 across the board.

Reproduction / evidence

Running the same 12-query trigger set against the same skill + description, only changing --num-workers:

Manually invoking one obvious positive with a single command file present shows the model's first tool call is Skill(<the skill>) — i.e. triggering works; only the parallel measurement is broken.

Fix

Isolate each query in its own project dir so a claude -p only ever sees its own command file:

import tempfile, shutil
iso_root = Path(tempfile.mkdtemp(prefix="trigeval-"))
project_commands_dir = iso_root / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"
try:
    project_commands_dir.mkdir(parents=True, exist_ok=True)
    command_file.write_text(command_content)
    # ... Popen(..., cwd=str(iso_root), env=env)
finally:
    shutil.rmtree(iso_root, ignore_errors=True)

This removes the cross-matching entirely (each worker's clean_name is the only skill in its tree) and lets parallel runs stay fast and correct.

Workaround until then

Pass --num-workers 1 to run_eval.py / run_loop.py. Note run_loop.py forwards --num-workers to run_eval, so the same bug silently corrupts the optimization signal at the default of 10 — every iteration optimizes against ~0 trigger rates.

Found while triggering-testing a new skill; happy to send a PR with the isolation fix if useful.