Group rows keep a 0.0 stderr when every subtask is entirely on the boundary, so the row with the most evidence carries no bound

What happens

When every subtask in a group sits entirely on 0 (or entirely on 1), the group row reports <metric>_stderr = 0.0, exactly as the task rows did before #4090. The group is the row with the most evidence behind it, so it is the row where a bound would be tightest, and it is the one row that ends up carrying no bound at all.

The case was raised by @buyan201430-code on #4090, measured on published result files. This is the group counterpart of #4070. #4090 fixes the task level and deliberately stops there: test_boundary_interval_does_not_reach_group_aggregation pins that exclusion, because the group has no score vector of its own, only pooled stderrs. This issue is for the remaining case.

Reproducible example

On current main (d6de8164), building a group from the seven real leaderboard_math_*_hard sizes (307, 123, 132, 280, 154, 193, 135; total n = 1324) with every subtask scoring 0 out of n:

SIZES = [307, 123, 132, 280, 154, 193, 135]
group = Group(name="leaderboard_math_hard",
              aggregate_metric_list=[AggMetricConfig(metric="exact_match", weight_by_size=True)])
for i, n in enumerate(SIZES):
    t = MockTask(f"sub{i}", agg={"exact_match": mean}, n_eval_docs=n)
    group.add(t)
    acc[f"sub{i}"] = _make_acc(t, {("exact_match", "none"): [0.0] * n})

n total                  : 1324
group exact_match,none   : 0.0
group exact_match_stderr : 0.0
group keys               : ['alias', 'exact_match,none', 'exact_match_stderr,none', 'name', 'sample_count', 'sample_len']

pooled_sample_stderr([0.0] * 7, SIZES) returns exactly 0.0: a subtask on the boundary contributes zero variance, so the pool of them is zero too.

Why it is worth a bound

A 95% Wilson bound at those sample sizes, from the helper #4090 adds:

After #4090 the subtask rows carry their bound and the group row does not, which reads backwards: the row pooling ten times more evidence is the least informative one on the page.

Prevalence

In the published result files (open-llm-leaderboard/results, revision aa81ecc38fdc5708254b833923368970efdf5ef5), a 600 file sample contains 634 rows where the metric is exactly 0.0 or 1.0 and the stderr is exactly 0.0. 94 of them (14.8%) are group rows, all leaderboard or leaderboard_math_hard. Script, file list and row level output: https://gist.github.com/Rodrigo-Palma/6c6cb3a3d3aac998a833e9545e0dafc6

That count deliberately excludes 49 files carrying the 2025-02-13 stamp, where the stderr is 0.0 even for scores that are not on the boundary. Those belong to #3966 and are a different thing: frozen published data, already not reproducible on main, which writes "N/A" when no stderr function is available. The case in this issue is the opposite situation, a score legitimately on the boundary and a stderr legitimately computed as 0.0, and it reproduces on main today.

Proposed rule

A group gets <metric>_boundary_ci95 only when every leaf subtask is itself entirely on the boundary and on the same side, computed over the summed n. Concretely:

• Descend to the leaves, not to the immediate children, so nested groups behave.
• All leaves all-zero, or all leaves all-one. A mix is not a degenerate vector and gets nothing.
• Any leaf off the boundary, or without a score vector, and the group gets nothing.
• Key and format identical to the task level, so a reader parses one shape in both places.

Two notes on the edges. On the boundary a weighted and an unweighted group mean coincide (every item is the same value), so the rule does not have to care which aggregation the group uses for the score itself. And the check has to key off the leaf vectors rather than the aggregated group cell: a group cell of 0.0 does not imply its children were on the boundary, so reading the cell alone could attach a <= 0.0029 bound to a group whose subtasks scored well above zero.

This depends on #4090, since it reuses the same boundary_ci helper, so it makes sense as a change on top of it rather than beside it. Happy to implement it if the approach looks right.