Marketing eval framework: expert-calibrated judge library in evals/ (across copy, creative, tool-calling)

Vision

Build an expert-calibrated marketing eval framework — a shared judge harness plus a growing library of judges that grade the quality of what a marketing agent produces and does, across every major marketing task: copywriting, creative, positioning, and tool-calling. Lives as a top-level evals/ framework in this repo, mapped 1:1 onto the skills.

Reverse-engineered from runneth-tools/hook-grader (MIT) — an open-source eval that grades whether an ad hook stops the scroll, calibrated against a world-class creative strategist (Alysha, Runneth/Motion). We can seed the ad-hook judge directly, with credit.

What hook-grader taught us (the transferable primitive)

An eval is just judge + answer key, plus machinery to measure the judge against the key. hook-grader is a clean 6-part instance:

• Judge (JUDGE.md) — a paste-able rubric (LLM system prompt) returning a structured JSON verdict (verdict, trigger, tactic, reasoning, improvement).
• Answer key (eval-set.json) — 116 examples hand-labeled by a named human expert, with a realistic distribution (73 FAIL / 43 PASS, not 50/50).
• Harness (eval-harness.mjs) — runs the judge over the key, reports agreement % + confusion breakdown (false-pass vs false-fail) + every disagreement.
• Grader CLI (grade.mjs) — grade new inputs at scale, BYO API key.
• Lib (lib.mjs) — provider-agnostic model calls (Anthropic/OpenAI auto-detect), batching, judge-loading.
• Methodology (METHODOLOGY.md) — calibration loop, blind two-judge consensus, what the score means.

Principles to carry over

1. An eval = judge + answer key. A judge is only trustworthy insofar as it agrees with human labels.
2. Rubric = failure taxonomy. The reliable signal in quality is negative — name the specific ways things fail; it's far easier and more reliable than predicting "great."
3. Calibration loop: run judge → find disagreements → sharpen rubric → repeat; publish the agreement number.
4. Match the real distribution in the answer key, or the judge learns the wrong prior.
5. Err strict for a bad-X detector — optimize precision on the fails ("kill dead-on-arrival, then test the rest"), not "talent scout for great."
6. Blind two-judge consensus (one model per lab, source hidden) for high-stakes grading.
7. Structured verdict with an improvement suggestion — the eval doubles as a coach.

The three eval types (marketing tasks have different shapes)

• Type A — Artifact quality (hook-grader is this). Grade a produced artifact against an expert-calibrated rubric. One judge + answer key per artifact. Covers most copywriting/creative/positioning outputs.
• Type B — Tool-calling / trajectory. Input = a task + the agent's tool-call trace; judge grades whether it picked the right tool, right args, right sequence, and used the result. Answer key = tasks with known-good trajectories. This is what "tool calling" needs — it is not artifact quality.
• Type C — Skill behavioral. Already exists as each skill's evals/evals.json (prompt + expected_output + assertions) — grades whether the agent behaves right (checks product-marketing.md, applies the framework, defers to the right sibling). The new framework is the quality/trajectory layer these don't cover; it complements, doesn't replace them.

Design note: keep the existing per-skill evals/evals.json (behavioral) as-is. The new top-level evals/ framework is a distinct layer. The README must make the two-layer distinction explicit so contributors don't conflate them.

Proposed structure

evals/
├── README.md              # the framework: eval = judge + answer key; the two layers; how to run & contribute
├── lib.mjs                # shared: provider-agnostic model calls, batching, agreement, confusion, consensus mode
├── harness.mjs            # generic: run ANY judge over ANY answer key → agreement % + confusion + disagreements
├── grade.mjs              # generic CLI: grade new inputs with any named judge (BYO key)
├── judges/
│   ├── ad-hooks/          # Type A — SEEDED from hook-grader (credit Alysha/Runneth, MIT)
│   │   ├── JUDGE.md
│   │   ├── answer-key.json
│   │   └── README.md
│   ├── headlines/         # Type A
│   ├── value-props/       # Type A
│   ├── ctas/              # Type A
│   ├── subject-lines/     # Type A
│   ├── landing-hero/      # Type A
│   └── tool-calling/      # Type B (trajectory) — its own harness variant
└── METHODOLOGY.md         # calibration loop, distribution, consensus, error-bias guidance

Generalize hook-grader's single-topic files into a judge-agnostic harness (harness.mjs judges/<name>), so adding a category = adding a judges/<name>/ folder with JUDGE.md + answer-key.json. lib.mjs stays provider-agnostic (Anthropic + OpenAI, model override via env), and defaults to current models (Opus/Sonnet 4.x) rather than hook-grader's pinned ones.

Judge ↔ skill mapping (starter set)

Answer-key & calibration methodology (per judge)

• 40–150 real, anonymized examples, hand-labeled by a credible human (name them, as hook-grader credits Alysha).
• Realistic distribution (most marketing artifacts fail the bar — reflect that).
• Calibration loop until agreement plateaus; publish the number + confusion split.
• Document the intended error bias (strict for bad-X detectors).
• Optional consensus mode for high-stakes categories.

Phased rollout

• Phase 0 — Harness. Generalize lib.mjs + harness.mjs + grade.mjs to be judge-agnostic. Seed judges/ad-hooks/ from hook-grader (credit). Reproduce the agreement number as the framework's first passing check. Write evals/README.md + METHODOLOGY.md.
• Phase 1 — Copywriting judges. headlines, value-props, CTAs, subject-lines — each with a hand-labeled answer key and a calibration pass.
• Phase 2 — Tool-calling (Type B). Define the trajectory answer-key format + harness variant; one worked judge (e.g. "did the agent choose the right analytics/attribution tool + args").
• Phase 3 — Broaden + contribute. More categories, consensus mode, a CONTRIBUTING-style guide so a new judge is a well-trodden path. Cross-link from each skill's README.

Strategic framing

This is an agent-native content play (see the pattern in memory): Runneth shipped an open eval as marketing. An open, expert-calibrated marketing eval framework is a category-defining asset for marketing-skills — more defensible than any single eval and the natural companion to the skills. MIT-friendly to seed (ad-hooks judge) with credit.

Open questions / decisions

• Human labelers: who calibrates each category's answer key? (For the seed, reuse hook-grader's labels with credit; net-new categories need a credible grader — us, or invited experts.)
• Runnable vs paste-only: ship the Node harness (BYO key) and the paste-the-JUDGE.md-into-any-AI path (hook-grader does both — keep both).
• CI: do we wire an agreement-threshold check into CI per judge, or keep it manual/opt-in (cost)?
• Naming: evals/ framework name and whether to also mirror it as a standalone public repo later for the marketing play.
• Scoring granularity: PASS/FAIL (hook-grader) vs a 1–5 score per category — decide per judge.

Credit / license

hook-grader is MIT. Seed the ad-hooks judge and reuse its answer-key labels with clear credit to Alysha (@alyshaboehmm) and Runneth, mirroring how we credit other reused OSS (gooseworks, kappaemme). Net-new judges are ours.

Scoped from a reverse-engineering pass on hook-grader. Build not started — this is the plan of record.