Live marketing-model benchmark (Opus/Sonnet/GPT/Kimi…) powered by the eval framework

Vision

A live-updating benchmark that ranks models — Opus, Sonnet, GPT, Kimi, Gemini, etc. — on marketing tasks, graded by the expert-calibrated judges from #479. Two payoffs from one system:

• Public leaderboard — "which model writes the best ad hooks / cold-email openers / headlines / positioning this month," always fresh, an authority/SEO asset and the ultimate agent-native content play.
• Internal model-routing table for Magister — the same scores tell Magister which model to actually use for each marketing sub-task (best, or best-per-dollar).

Depends on #479 (the OSS evals/ framework), which is the engine + credibility layer. This issue is the flagship application of it.

The critical split: generator vs judge

Two roles a model plays — conflating them kills the benchmark:

• Generator (the contestant) — each model writes the hooks/headlines/emails. This is what's ranked.
• Judge (the grader) — a fixed, neutral, blind rubric-model scores everyone's output.

If a contestant grades its own category you get self-preference bias (models rate their own outputs higher). The judge must be held constant, blind to authorship, and ideally a consensus of two judges from different labs. hook-grader already names this failure mode ("rules out 'you just picked a model that likes your own hooks'").

Pipeline

1. Task set — 50–100 held-out briefs across categories (e.g. "write a scroll-stopping hook for [anonymized product X]"). Rotating, not public.
2. Contestants — Opus 4.8 · Sonnet 4.6 · GPT-x · Kimi · Gemini · … Each generates outputs for every task under identical conditions (same prompt, temp, N samples per task).
3. Judging — the calibrated judge (blind, order-randomized) grades every output PASS/FAIL or 1–5; consensus of 2 neutral judges.
4. Aggregate — per model × per category: pass-rate / avg score → leaderboard.
5. Live update — cron reruns on a schedule and when a new model drops (new model = add a contestant; board refreshes).

Why calibrated judges = defensibility

Anyone can ask a model to rank outputs — but that's one model's taste, and circular when that model is also a contestant. The #479 judges are proven to agree with a world-class human expert (~74% for hooks). So the board is "ranked by a rubric that demonstrably matches expert human judgment, blind and by consensus," not a popularity contest. The published agreement number per judge is the credibility.

Integrity design (the actual work)

• Blind + neutral + consensus judging — judge never sees the model name; 2 judges from different labs; a contestant never judges its own category.
• Held-out, rotating task sets — publish the calibration answer keys (they prove the judge), keep the scoring briefs fresh/hidden or models get tuned to them.
• Pinned judge version — grader model + rubric version frozen; upgrading the judge triggers a full re-baseline so score movements mean "models changed," not "the ruler changed."
• Controlled generation — identical prompt/temp, randomized order, length-normalized, N samples per task (one output isn't a score).
• Calibration floor — a judge under ~70% human agreement isn't trusted to rank models.

Two grading loops (don't conflate)

• Per-model, scheduled (this issue) — the benchmark/leaderboard. Runs on a cron across contestants.
• Per-artifact, at generation (telemetry) — every artifact Magister generates gets graded by the relevant judge, stored, surfaced as a live quality dashboard, and gated (a failing hook doesn't auto-publish). Sits alongside Magister's existing run_aeo_audit / get_ai_visibility_set scores as a new "Creative/Copy Quality" score. Cost-sensitive → sample.

The routing payoff

The board will almost certainly show no single winner — likely "Kimi tops hooks, Opus tops positioning, Sonnet tops cold-email at a fraction of the cost." Magister routes each marketing sub-task to the model that scores best (or best-per-dollar) on that category. So the leaderboard is simultaneously (a) a public authority asset and (b) the routing table that makes Magister's own output better — a two-for-one.

Surfaces in Magister

• Public leaderboard page — per-category model rankings, methodology/agreement numbers, "updated [date]," auto-refreshing.
• Internal — model-routing config driven by category scores; per-brand/project quality telemetry.

Data model sketch

• judges (from #479): name, type, rubric version, human-agreement %.
• tasks: id, category, brief, held-out flag, rotation cohort.
• contestants: model id, provider, cost/token, active.
• generations: task_id, model_id, sample_idx, output, ts.
• scores: generation_id, judge_id (×consensus), verdict/score, reasoning, ts.
• Leaderboard = aggregate of scores by model × category over a window.

Cost considerations

N models × M tasks × S samples × K judges (×2 consensus) is real token spend, but it's scheduled, not per-request. Control via: sampling, cadence (weekly baseline + on-new-model), and cheaper judge models where the calibration number holds.

Open questions / decisions

• Judge model(s): which 2 neutral labs for consensus, and how to stay neutral as contestants (some judges are also contestants elsewhere).
• Scoring: PASS/FAIL vs 1–5 vs pairwise/Elo (LMArena-style head-to-head may be more robust than absolute PASS-rate for ranking).
• Cadence & cost ceiling for the scheduled runs.
• Held-out rotation mechanics (how often, how sourced).
• Public vs gated leaderboard; how much methodology to expose.
• Category coverage at launch — start with hooks (seeded) + 2–3 copy categories, expand as #479 adds judges.

Dependencies

• #479 must produce at least one calibrated judge first (ad-hooks seed). This benchmark is Phase 2+ of that effort — the flagship application, not a prerequisite.

---

Scoped from the eval-framework discussion. Primary purpose clarified by Corey: evaluating different models (Opus vs Sonnet vs GPT vs Kimi…) on marketing tasks. Build not started — plan of record. Companion to #479.