At 0 (or 100%) accuracy the reported stderr is exactly 0.0 at any N, so null results claim infinite precision; a boundary-aware interval or flag would fix it

Summary

mean_stderr returns 0.0 for an all-zero score vector at any sample size, and bootstrap_stderr agrees, because every resample of zeros is zeros. End to end, a model that scores 0 on a task is reported as 0.0 +- 0.0000: a 25-question run and a 5000-question run claim identical, infinite certainty that the capability is absent. The same holds symmetrically at perfect scores. This is expected behavior of CLT/Wald standard errors at the boundary, but the harness's numbers are quoted directly in model cards and leaderboards, where +- 0.0000 reads as precision rather than as a degenerate estimator.

Relation to open issues

#3966 documents stderr = 0.0 for scores strictly between 0 and 1 and correctly concludes those records are internally inconsistent; its analysis notes in passing that stderr can only legitimately be 0 "when the score is exactly 0 or exactly 1". This report is about that remaining, legitimate case: the value is internally consistent and still the wrong thing to publish, because at the boundary the estimator degenerates exactly where the claim (capability absent, or saturated) is strongest. #4017 and #4019 propose Wilson CIs at the output layer; the boundary case here is arguably the sharpest argument for them, since it is where the missing interval is not merely narrow but absent entirely.

Reproduction

lm-eval 0.4.12 (release) and 0.4.13.dev0 (current main), Python 3.13.14, macOS. Unit level, identical on both versions:

End to end:

res = lm_eval.simple_evaluate(model="dummy", tasks=["gsm8k"], limit=25, random_seed=0)
# results.gsm8k: exact_match 0.0, exact_match_stderr 0.0 (both filters)

Why this matters

1. The null result is the case that matters for capability rule-outs, and it is the one case where the reported uncertainty is maximally wrong: the true 95% upper bound at 0/25 is about 13.7% (Clopper-Pearson) or 9.5% (Jeffreys), not 0.
2. Sample size becomes invisible exactly at the boundary. Downstream readers cannot distinguish a cheap null from an exhaustive one, though the two support very different claims.
3. Aggregations inherit the zero: pooled and combined stderrs over subtasks treat a boundary subtask as contributing zero variance, tightening group intervals.

Possible directions

• Report a one-sided Jeffreys or Wilson bound when a metric sits exactly on a boundary (the rule of three, 3/N, is the crude version and costs one line). If the Wilson CI proposal in #4017/#4019 lands, the boundary case comes for free and this issue reduces to a test case for it.
• Or leave the stderr as is but emit a boundary flag column so tables can render 0.0 (<= 0.095 at 95%) instead of 0.0 +- 0.0.
• Either change is metric-layer only and touches no task definitions.

Happy to PR either variant with tests if there is appetite.