Shared tasks are double-counted in nested group aggregation

Author: sdivyanshu90Created Sep 16, 2026Updated Sep 16, 2026

Problem

A hierarchical group can contain sibling subgroups that reference the same leaf task. The task is evaluated once, but the parent group's aggregate counts its result once for every path through the hierarchy.

For example:

root
├── subgroup_a
│   └── shared_task
└── subgroup_b
    ├── shared_task
    └── other_task

Suppose the task results are:

Task Samples acc,none
shared_task 100 0.0
other_task 200 1.0

The parent currently reports sample_len=400 and acc,none=0.5. It should report sample_len=300 and a size-weighted accuracy of 2 / 3.

Reproduction

python
shared = MockTask("shared_task")
other = MockTask("other_task")

subgroup_a = Group("subgroup_a")
subgroup_a.add(shared)

subgroup_b = Group("subgroup_b")
subgroup_b.add(shared)
subgroup_b.add(other)

root = Group(
    "root",
    aggregate_metric_list=[
        AggMetricConfig(metric="acc", filter_list=["none"])
    ],
)
root.add(subgroup_a)
root.add(subgroup_b)

metrics = {
    "shared_task": {"sample_len": 100, "acc,none": 0.0},
    "other_task": {"sample_len": 200, "acc,none": 1.0},
}

print([task.task_name for task in root.get_all_tasks()])
print(root.aggregate(metrics))

Current output:

['shared_task', 'shared_task', 'other_task']
{'sample_len': 400, 'acc,none': 0.5, ...}

Cause

Group.get_all_tasks() recursively concatenates each subgroup's leaves. It does not track task names already reached through another branch. Group.aggregate() then uses that repeated list for:

  • the group-level sample_len;
  • the values and sizes passed to metric aggregation;
  • per-metric sample counts;
  • filter discovery and missing-metric reporting.

Evaluation results are stored by task name, so the second path does not represent a second evaluation. It only reuses and reweights the same result.

The duplicate check in TaskManager prevents one task from appearing in separate top-level selections, but it treats all leaves under one top-level group as the same source. A shared leaf under sibling subgroups therefore passes validation and reaches aggregation.

Impact

User-defined hierarchical benchmark groups can publish incorrect parent scores and sample counts. A shared task receives extra weight according to the number of subgroup paths that reach it, so changing hierarchy alone can change the benchmark result without changing the evaluated tasks.

No built-in group currently appears to use this diamond-shaped hierarchy, but the public group configuration and API allow it.

Expected behavior

A parent aggregate should count each task result once per unique task name. Sibling subgroups may still expose their own rows, but their shared leaf must not receive multiple weight in their common parent.

Proposed fix

Make recursive leaf discovery de-duplicate by task_name while preserving first-seen traversal order. This fixes every consumer of the leaf list consistently, while retaining the existing error for overlap between separately requested top-level groups.

Add regression coverage for:

  1. unique recursive leaf discovery through sibling subgroups;
  2. the resulting group sample_len, per-metric sample count, and size-weighted score.

Source: EleutherAI/lm-evaluation-harness