Shared tasks are double-counted in nested group aggregation
Problem
A hierarchical group can contain sibling subgroups that reference the same leaf task. The task is evaluated once, but the parent group's aggregate counts its result once for every path through the hierarchy.
For example:
root
├── subgroup_a
│ └── shared_task
└── subgroup_b
├── shared_task
└── other_taskSuppose the task results are:
| Task | Samples | acc,none |
|---|---|---|
shared_task |
100 | 0.0 |
other_task |
200 | 1.0 |
The parent currently reports sample_len=400 and acc,none=0.5. It should report sample_len=300 and a size-weighted accuracy of 2 / 3.
Reproduction
shared = MockTask("shared_task")
other = MockTask("other_task")
subgroup_a = Group("subgroup_a")
subgroup_a.add(shared)
subgroup_b = Group("subgroup_b")
subgroup_b.add(shared)
subgroup_b.add(other)
root = Group(
"root",
aggregate_metric_list=[
AggMetricConfig(metric="acc", filter_list=["none"])
],
)
root.add(subgroup_a)
root.add(subgroup_b)
metrics = {
"shared_task": {"sample_len": 100, "acc,none": 0.0},
"other_task": {"sample_len": 200, "acc,none": 1.0},
}
print([task.task_name for task in root.get_all_tasks()])
print(root.aggregate(metrics))Current output:
['shared_task', 'shared_task', 'other_task']
{'sample_len': 400, 'acc,none': 0.5, ...}Cause
Group.get_all_tasks() recursively concatenates each subgroup's leaves. It does not track task names already reached through another branch. Group.aggregate() then uses that repeated list for:
- the group-level
sample_len; - the values and sizes passed to metric aggregation;
- per-metric sample counts;
- filter discovery and missing-metric reporting.
Evaluation results are stored by task name, so the second path does not represent a second evaluation. It only reuses and reweights the same result.
The duplicate check in TaskManager prevents one task from appearing in separate top-level selections, but it treats all leaves under one top-level group as the same source. A shared leaf under sibling subgroups therefore passes validation and reaches aggregation.
Impact
User-defined hierarchical benchmark groups can publish incorrect parent scores and sample counts. A shared task receives extra weight according to the number of subgroup paths that reach it, so changing hierarchy alone can change the benchmark result without changing the evaluated tasks.
No built-in group currently appears to use this diamond-shaped hierarchy, but the public group configuration and API allow it.
Expected behavior
A parent aggregate should count each task result once per unique task name. Sibling subgroups may still expose their own rows, but their shared leaf must not receive multiple weight in their common parent.
Proposed fix
Make recursive leaf discovery de-duplicate by task_name while preserving first-seen traversal order. This fixes every consumer of the leaf list consistently, while retaining the existing error for overlap between separately requested top-level groups.
Add regression coverage for:
- unique recursive leaf discovery through sibling subgroups;
- the resulting group
sample_len, per-metric sample count, and size-weighted score.
Source: EleutherAI/lm-evaluation-harness