claude-api evals: 1/sqrt(n·reps) treats reps as extra cases, so the noise floor is understated at higher rep counts

The eval guides added in #1930 (skills/claude-api/shared/evals/) size the noise floor as 1/sqrt(n·R) (n cases, R reps) and treat cases and reps as interchangeable:

• eval-audit.md §5: "Cases and reps are two knobs on the same dial", with more reps listed as the cheapest lever
• eval-hillclimb.md Step 3: "reps and test-size are two knobs on the same dial"
• build-eval.md (set sizing): "fifty-plus inputs with a random held-out slice, or fewer inputs with more reps, are two routes to the same resolution"

Why that doesn't hold

Reps re-sample runs, not cases. With n cases and R reps per case, the interval over per-case results has a half-width of about 1.96 × sqrt((s_case² + s_run²/R) / n). Reps shrink only the run-to-run term, so the interval levels off at 1.96 × s_case / sqrt(n) however many reps are added. 1/sqrt(n·R) matches only when every case has the same pass rate, and for a paired difference it can be up to about √2 too narrow even then, because both arms are re-run.

report/SCHEMA.md already specifies the right interval: PairedDelta is a Wald CI over per-case deltas, so n points, not n × R. As R grows, the guides' rule and the report's own interval move apart. Miller, "Adding Error Bars to Evals" (2024, §3.1) makes the same point about resampling: once the within-question term is small next to the between-question variance, more samples barely move the standard error.

What it does in practice

25 cases with pass rates spread from 5% to 95%, a change worth +10 points on average that helps some cases more than others (per-case effect sd 0.15), 20,000 simulated evals. "Covers" is how often the 95% interval contains the average effect on new cases:

Following the "more reps, cheapest" advice from 2 to 50 reps, the user is told the floor is ±2.8 points, the report's own interval comes back near ±6, and the guide's figure holds 62% of the time. The single split score in eval-hillclimb.md behaves the same way: at 10 reps the 1/sqrt(n·reps) interval covers 71%, at 50 reps 40%.

If the question were only "did it improve on these exact 25 cases", the guide's figure would be close. The guides use the floor for deploy decisions and held-out splits, though, so the target is new inputs.

import numpy as np
rng = np.random.default_rng(0)
n, sims = 25, 20000
draw = lambda size: (rng.uniform(0.05, 0.95, size), 0.10 + 0.15 * rng.standard_normal(size))
pA, eff = draw(10**6); true = np.mean(np.clip(pA + eff, 0, 1) - pA)  # average effect on new cases
for R in (2, 10, 50):
    pA, eff = draw((sims, n))  # cases differ in difficulty and in how much the change helps
    pB = np.clip(pA + eff, 0, 1)
    d = rng.binomial(R, pB) / R - rng.binomial(R, pA) / R  # per-case delta, R reps per arm
    guide = 1 / np.sqrt(n * R)
    wald = 1.96 * d.std(1, ddof=1) / np.sqrt(n)  # SCHEMA.md PairedDelta
    cov = lambda hw: np.mean(abs(d.mean(1) - true) <= hw)
    print(f"R={R:2d}  guide ±{100*guide:.1f} covers {cov(guide):.0%}   per-case ±{100*wald.mean():.1f} covers {cov(wald):.0%}")

Suggested fix

I'll open a docs-only PR (five lines across the three files) that:

• uses the measured 1.96 × sd(per-case deltas) / sqrt(n) once a variant has run against the baseline
• keeps 1/sqrt(n·R) as a rough sketch for a single pass-rate before then
• gives the split range (±14 to ±20 points at 25 cases × 2 reps, near a 50% pass rate)
• says reps help only while run-to-run noise outweighs the case-to-case spread

Would you take that wording, or would you prefer a different fix?

Disclosure: I maintain Driftproof, an evaluation tool. This came from a direct read of shared/evals/ and report/SCHEMA.md and the reproduction above, not from any tool.