Drift data for 3 of these skills across the sonnet-4-6 → sonnet-5 release
Hi Addy — I run a small OSS project (Driftproof) that re-tests agent skills across model releases: same suite, with/without the skill, judge sampled 5x so every score carries a confidence band, and a regression is only claimed when bands separate AND the move clears a 0.05 effect floor.
For our first report I wrote eval suites for three skills from this repo (code-review-and-quality, git-workflow-and-versioning, documentation-and-adrs), each case grounded in a claim the skill's own SKILL.md makes, and ran them across the sonnet-4-6 → sonnet-5 release. Results, with receipts:
- code-review-and-quality: WITHIN NOISE (held up)
- git-workflow-and-versioning: 1 case regressed (semver-clean-bump)
- documentation-and-adrs: MIXED (one sharp regression, one improvement)
Full tables + methodology: https://driftproofhq.com/reports/001/ The suites themselves: https://github.com/driftproofhq/driftproof/tree/main/suites
Offer, take or leave any of it: the suites are Apache-2.0 and yours to adopt upstream if useful; npx driftproof run re-runs them on any model with your own key; and there's a GitHub Action + badge if continuous verification on releases is interesting. If you think any case tests something your skill doesn't actually claim, I'd genuinely like to know — suite fairness is the part I care most about getting right.
Source: addyosmani/agent-skills