[skill-creator] Trigger-eval harness silently broken on Windows: select() on pipes, probe files leak into global ~/.claude/commands, parallel twins cap recall at ~1/N

Summary

The skill-creator description-optimization harness (skills/skill-creator/scripts/run_eval.py + run_loop.py) is structurally broken on Windows. Worse than crashing, it fails silently: every probe dies or mismeasures, the loop reports recall=0% (or a hard ~1/N cap), and then happily "improves" the description against pure noise for up to 5 iterations. Each probe is a full claude -p session that loads the user's entire MCP/skills context, so a user running the loop as documented burns a large share of their plan's usage limit on measurements that were never valid.

All four defects below were hit in one real session (Claude Code 2.1.247 desktop, Windows 11 Pro, Python 3.14.6, anthropic-skills plugin build) and confirmed present in this repo at current main.

Defect 1 — select.select() on a subprocess pipe → every probe dies with WinError 10038

run_eval.py:108:

ready, _, _ = select.select([process.stdout], [], [], 1.0)

On Windows, select() supports sockets only. Every single probe raises [WinError 10038] An operation was attempted on something that is not a socket, gets swallowed as Warning: query failed, and counts as "not triggered". The loop then prints a clean-looking precision=100% recall=0% table and proceeds to "improve" the description.

Fix that worked locally: replace the select loop with a daemon reader thread pumping process.stdout.read1(8192) into a queue.Queue, and queue.get(timeout=1.0) in the main loop. Cross-platform, keeps the early-kill-on-trigger behavior.

Defect 2 — find_project_root() walks up to $HOME and writes probe files into the user's global ~/.claude/commands

run_eval.py:22 walks up from Path.cwd() until it finds a .claude directory. The documented invocation (cd <skill-creator dir> && python -m scripts.run_loop ...) starts from a plugin directory with no .claude, so on a default setup the walk reaches the user's home — which does contain ~/.claude — and run_eval.py:53 then writes every probe command file into the user's global ~/.claude/commands/.

Consequences observed:

• Probe skills (<name>-skill-<uuid>) appeared in the available-skills list of every other session on the machine, including unrelated interactive ones.
• A killed/interrupted run never reaches the finally cleanup: 10 orphaned probe command files were left permanently in the global commands dir.

Defect 3 — parallel workers expose N twin copies to every probe → recall is capped at ~1/N

run_eval.py:198 runs probes through ProcessPoolExecutor (default ~10 workers). Because all workers share the same project root (Defect 2 makes it global), each probe session sees ~10 identical skills with different UUID-suffixed names and invokes an arbitrary one — while the harness only counts a match on its own UUID name.

Measured effect: two consecutive iterations with materially different descriptions both scored exactly recall=11% train / 8% test (~1/10 — collision probability, not signal). After isolating each probe in its own throwaway root, the same description scored 60/60.

Fix that worked locally: in run_single_query, ignore the shared project_root argument and use tempfile.mkdtemp(prefix="skill-trigger-") per probe, creating .claude/commands/ inside it, with shutil.rmtree(project_root, ignore_errors=True) in finally. This also fixes Defect 2's leak and orphaned files.

Defect 4 — read_text() without encoding → crash on non-Latin eval sets

run_loop.py:261:

eval_set = json.loads(Path(args.eval_set).read_text())

On Windows this decodes with cp1252 and immediately raises UnicodeDecodeError for any UTF-8 eval set containing non-Latin queries (Arabic in our case — exactly the multilingual eval queries the skill's own docs recommend). Same pattern exists for other read_text()/write_text() calls. Workaround: PYTHONUTF8=1; proper fix: encoding="utf-8" everywhere.

Repro

1. Windows 11, Claude Code CLI ≥ 2.1.x logged in, Python 3.13+.
2. Create any eval set with Arabic queries → Defect 4 crash.
3. Retry with PYTHONUTF8=1 → Defect 1: wall of WinError 10038 warnings, recall=0%.
4. Patch Defect 1, hide nothing else → watch probe files appear in ~/.claude/commands (Defect 2) and recall plateau at ~1/workers regardless of description (Defect 3).

Suggested upstream fixes

• Thread-pump (or asyncio with Proactor) instead of select() on pipes.
• Per-probe isolated temp project root, cleaned in finally (fixes 2 + 3 together).
• Explicit encoding="utf-8" on all text I/O.
• A loud failure when >50% of probes error out — a harness that can't measure should stop the loop, not feed recall=0% into the improver and spend the user's quota on noise.

Happy to open a PR with the three local patches if useful — they took the harness from 100% invalid probes to a clean 60/60 run on the same machine.