skill-creator: silent benchmark failures (layout mismatch, inverted delta), broken trigger evals on Windows, and skill shadowing in trigger evals

Six reproducible issues in skills/skill-creator, found while auditing and testing the skill end-to-end. Roughly in order of impact:

1. SKILL.md directory layout is silently incompatible with aggregate_benchmark.py

SKILL.md instructs subagents to save to <workspace>/iteration-<N>/eval-<ID>/<config>/outputs/ and to put grading.json "in each run directory" — a flat layout with no run-N level. aggregate_benchmark.py however skips every config dir without run-* subdirectories (if not list(config_dir.glob("run-*")): continue) and only reads eval-*/<config>/run-*/grading.json.

Repro: build a fixture in the exact SKILL.md layout, run python -m scripts.aggregate_benchmark <dir>. Result: benchmark.json with runs: [], summary prints only Delta: +0.00, exit code 0. Anyone following SKILL.md literally gets an empty benchmark with no error, and the iteration decision is made on "0% pass rate" noise. (The eval viewer, by contrast, handles the flat layout fine.)

Suggested fix: treat a config dir containing grading.json or outputs/ but no run-* as a single run (run_number = 1), and exit non-zero when zero runs were found instead of writing an empty benchmark.

2. Delta sign flips in the improve flow (old_skill baseline)

SKILL.md's improve flow uses old_skill/ as baseline next to with_skill/. aggregate_benchmark.py discovers configs alphabetically (sorted(eval_dir.iterdir())) and computes delta = configs[0] - configs[1]. Since old_skill sorts before with_skill, the baseline becomes primary.

Repro: with_skill pass rate 1.0, old_skill 0.0 → output Delta: -1.00. A clear improvement reads as a regression in benchmark.json, benchmark.md and the console summary. (The standard flow is unaffected only by the accident that with_ sorts before without_.)

Suggested fix: fixed priority (with_skill/new_skill primary; without_skill/old_skill baseline) instead of discovery order. Also: SKILL.md line "Put each with_skill version before its baseline counterpart" is not actionable (the script decides), and schemas.md only allows with_skill|without_skill for configuration while SKILL.md mandates old_skill — worth reconciling.

3. eval_name is documented but never propagated

schemas.md documents runs[].eval_name in benchmark.json and viewer.html displays it as the eval heading, but aggregate_benchmark.py reads only eval_id from eval_metadata.json. The Benchmark tab therefore always shows "Eval 0" even though SKILL.md explicitly asks authors to give evals descriptive names.

Suggested fix: read eval_name from eval_metadata.json and include it in each runs[] entry.

4. run_eval.py: select.select() on pipes crashes every worker on Windows — and the failure is silent

run_single_query polls the claude -p subprocess with select.select([process.stdout], ...). On Windows, select() only supports sockets; the call raises OSError: [WinError 10093] immediately. The parent catches the exception per query and records False ("not triggered"). Net effect on Windows: every should-trigger query scores 0, every should-not-trigger query "passes", exit code 0, and run_loop.py optimizes the description against garbage — the only hint is a stderr warning per run.

Suggested fix: replace select with a platform-neutral reader thread (blocking read1() into a queue.Queue, main loop polls with timeout) — patch available; happy to open a PR. Alternatively, abort the eval on such errors instead of recording False.

5. Trigger evals: an installed copy of the skill shadows the synthetic test skill

run_eval.py creates a synthetic command (<name>-skill-<uuid>) carrying the candidate description and counts a trigger only when that name is invoked. But if the skill under test is also installed in user scope (~/.claude/skills/<name> — the normal case when improving one's own installed skill), the real skill appears in the subprocess's skills list and wins: captured stream-json shows the model invoking Skill: {"skill": "<name>"} (the real one), which the eval counts as "not triggered". All positive queries then read as false negatives and the optimization loop measures nothing.

Suggested fix: at minimum warn when ~/.claude/skills/<name> exists and advise temporarily moving it; better, fail fast. (Detecting the real-skill invocation and counting it as a trigger would be wrong — it fires on the installed description, not the candidate.)

6. No encoding="utf-8" anywhere — cp1252 pitfalls for non-ASCII skills on Windows

None of the read_text()/write_text()/open() calls in scripts/ and eval-viewer/ pass an encoding. On Windows (default cp1252, UTF-8 mode off) this means: UTF-8 SKILL.md files with umlauts are silently read as mojibake (für → für) and fed into trigger tests; emoji in SKILL.md raise UnicodeDecodeError; the HTML report write in run_loop.py crashes with UnicodeEncodeError on the ✓/✗ characters after the (expensive) eval iterations have already run; package_skill.py's emoji prints crash under piped stdout. Workaround: PYTHONUTF8=1. Fix: pass encoding="utf-8" throughout (and consider sys.stdout.reconfigure(errors="replace") for the emoji prints).

---

Happy to provide the working patches for 1-5 as PRs (they are running locally: flat-layout support + loud zero-run failure, fixed delta priority, eval_name pass-through, thread-based reader, shadowing warning).