writing-skills: GREEN measures compliance, not outcome

Summary

writing-skills defines GREEN as the agent complying with the skill. For discipline-enforcing skills that is exactly right — compliance is the outcome. For skills that produce an artifact (a document, a config, a fix), compliance is a proxy, and I hit a case where the proxy passed cleanly while the outcome went entirely unmeasured.

Current text:

TDD Concept	Skill Creation
Test passes (GREEN)	Agent complies with skill present

and the per-type criteria: "Agent follows rule under maximum pressure", "Agent successfully applies technique to new scenario", "Agent correctly identifies when/how to apply pattern", "Agent finds and correctly applies reference information". Every one observes the agent. None observes whether what the agent produced did the thing the skill exists for.

Evidence

I used writing-skills to build a skill for revising the prompts of a deployed multi-component LLM system — Iron Law, baseline first, the "Match the Form to the Failure" table, all of it. Then I ran a controlled trial on one real defect: three agents with the skill, three without, identical codebase state, isolated worktrees, one variable.

The skilled arm complied on essentially every process criterion — 3/3 diagnosed the root cause correctly (the unskilled arm: 0/3), 3/3 handled version immutability correctly (1/3), 3/3 named the trade-off and reported their own cost (0/3).

The single agent that produced evidence the fix worked had no skill at all. It ran the real model six times against the old version and six against the new, showed the defect reproducing and then gone, and quoted the outputs. None of the three skilled agents did this. They produced careful, well-reasoned arguments instead.

By this skill's own GREEN criterion, my skill passed. The one thing that was not a proxy went unmeasured — and worse, none of the three said "this is unverified", so a reader could not tell an argued change from a tested one.

Proposed patch

1. TDD mapping table — split the GREEN row:

| **Test passes (GREEN)** | Agent complies with skill present |

becomes

| **Test passes (GREEN), compliance** | Agent complies with skill present |
| **Test passes (GREEN), outcome** | The artifact the agent produced achieves what the skill exists for — measured, not argued. Compliance without this is a proxy. |

2. Testing All Skill Types — for Technique / Pattern / Reference skills, add an outcome criterion alongside the existing one, e.g.:

**Success criteria:** Agent successfully applies technique to new scenario
**Outcome criteria:** The result of applying it is measurably better than
the baseline result. If it cannot be measured, the skill must require the
agent to say so explicitly rather than stay silent.

3. Common Rationalizations for Skipping Testing — add:

Excuse	Reality
"The agent followed the skill, so the skill works"	Compliance is not outcome. Measure what the artifact did.
"The reasoning was careful, so the result is sound"	Careful reasoning is a hypothesis. Run it.

Why I think it matters beyond my case

The failure mode is the same one the wider literature on self-improving agents keeps hitting: an optimizer scored by a judge that has not earned its authority to say whether the system improved. A skill graded on whether agents obey it is graded by something correlated with, but not identical to, the thing anyone cares about. Cheap to close, and closing it also gives skill authors a reason to build the measurement harness at all.

Happy to send this as a PR instead if you'd prefer — I kept it as an issue since it touches the framing rather than just the wording.