aggregate_reports drops payload_init and tree_data rows, so aggregated reports lose payload provenance and tree search history

Description

python -m garak.analyze.aggregate_reports silently drops two kinds of data that garak's own post-run tools read.

The aggregator copies only the entry types it explicitly whitelists. Two written entry types have live readers but are not on that list:

• payload_init — one row per payload corpus loaded (garak/payloads.py:106). report_digest renders them as the report's payload list, and aggregate_reports.main() rebuilds the digest from the aggregated file, so the aggregated report ends up claiming no payload corpora were used at all.
• tree_data — one row per node explored by a TreeSearchProbe (garak/probes/base.py:634). Its only reader is garak/analyze/get_tree.py:31, so python -m garak.analyze.get_tree on an aggregated report prints "No tree data in output report JSONL" for a run that did explore a tree: the search history is gone.

Both fail silently: the aggregator exits 0 and warns nothing.

Steps to reproduce

Offline, no third-party plugins, from a checkout at 8d1259ef310e4803cf5a4cc77267fdfdc24434ec:

python -m garak --target_type test.Repeat -p badchars.BadCharacters -d always.Pass -g 1 --report_prefix /tmp/pay1
python -m garak.analyze.aggregate_reports -o /tmp/agg.jsonl /tmp/pay1.report.jsonl

badchars.BadCharacters loads the harmful_behaviors payload, so the run's own report contains it:

single report kinds: {'start_run setup': 1, 'init': 1, 'payload_init': 1, 'plugin_cache': 1,
                      'attempt': 512, 'eval': 1, 'probe_summary': 1, 'completion': 1, 'digest': 1}

build_digest('/tmp/pay1.report.jsonl')['meta']['payloads']
  -> ["harmful_behaviors  {'entries': 15, 'payload_name': 'harmful_behaviors', ...}"]

The aggregate loses it, and so does the rebuilt digest:

aggregate kinds: {'start_run setup': 1, 'init': 1, 'plugin_cache': 1,
                  'attempt': 256, 'eval': 1, 'probe_summary': 1, 'digest': 1}     # no payload_init

build_digest('/tmp/agg.jsonl')['meta']['payloads'] -> []

Note that this already happens with a single input file, so it is not an edge case of merging conflicting runs.

Cause

garak/analyze/aggregate_reports.py:31-37:

        if entry["entry_type"] not in (
            "attempt",
            "eval",
            "probe_summary",
            "plugin_cache",
        ):
            continue

The writer is garak/payloads.py:103-116 ("entry_type": "payload_init", with payload_name, payload_path, entries, filesize, mtime). The reader is garak/analyze/report_digest.py:79-84, which appends one entry per payload_init row into the digest's meta.payloads. And aggregate_reports.py:181 (at 8d1259ef) calls report_digest.build_digest(a.output_path) on the aggregated file and appends that fresh digest, so nothing downstream can recover what was filtered out. The whitelist already carries plugin_cache, another metadata row the digest consumes, so the omission looks like an oversight rather than a decision.

Expected behavior

Payload rows should pass through the aggregator like the other carried entry types, so the aggregated report's payload section lists the corpora the contributing runs actually loaded.

Test case

Per contributing guidance, the failing case. Adding these to tests/analyze/test_aggregate.py and running on 8d1259ef gives 2 failures; the existing 5 tests in that file keep passing because no committed asset contains a payload_init row:

PAYLOAD_ROW = {
    "entry_type": "payload_init",
    "loading_complete": "payload",
    "payload_name": "test_payload",
    "payload_path": "/tmp/test_payload.json",
    "entries": 3,
    "filesize": 128,
    "mtime": "1700000000.0",
}


def _report_with_payload(tmp_path):
    source = Path(__file__).parents[1] / "_assets" / "analyze" / "test.report.jsonl"
    lines = source.read_text(encoding="utf-8").splitlines()
    lines.insert(2, json.dumps(PAYLOAD_ROW))
    report = tmp_path / "payload.report.jsonl"
    report.write_text("\n".join(lines) + "\n", encoding="utf-8")
    return str(report)


def test_aggregate_keeps_payload_rows(tmp_path):
    """Aggregation must carry payload provenance through."""
    from garak.analyze.aggregate_reports import main as aggregate_main

    aggregated = str(tmp_path / "agg.report.jsonl")
    aggregate_main(["-o", aggregated, _report_with_payload(tmp_path)])

    entry_types = [
        json.loads(line)["entry_type"]
        for line in open(aggregated, encoding="utf-8")
        if line.strip()
    ]
    assert entry_types.count("payload_init") == 1, (
        "the payload_init row must survive aggregation, not be filtered out as an"
        f" unlisted entry type (got {entry_types.count('payload_init')})"
    )


def test_aggregated_digest_names_the_payload(tmp_path):
    """The digest rebuilt for an aggregated report must still list the payloads."""
    from garak.analyze.aggregate_reports import main as aggregate_main

    aggregated = str(tmp_path / "agg.report.jsonl")
    aggregate_main(["-o", aggregated, _report_with_payload(tmp_path)])

    digest = garak.analyze.report_digest.build_digest(aggregated)
    payloads = digest["meta"]["payloads"]
    assert any("test_payload" in p for p in payloads), (
        "aggregated report claims no payload corpora were loaded:"
        f" meta.payloads is {payloads}"
    )

Measured on 8d1259ef: test_aggregate_keeps_payload_rows → AssertionError: … got 0; test_aggregated_digest_names_the_payload → AssertionError: … meta.payloads is []. Two lines in _process_file_body (the import of Path plus the whitelist entry) make both pass.

Open question

Aggregating N reports yields N payload_init rows and N entries in meta.payloads. That matches how plugin_cache rows are already carried (one per input file), and it is not a new shape for the report: the committed garak-report/extracted_digest.json already holds 151 payload strings for 14 unique names (python_code_execution ×59, sql_injection ×31, web_html_js ×17), because a corpus is re-recorded every time it loads, and the viewer normalises and de-duplicates by name (garak-report/src/hooks/usePayloadParser.ts:96,144-151). report_digest's plugin-cache merge does merge, so if payload merging belongs in _parse_report rather than the viewer, say so and I will add it in the same PR.

Duplicate check

• gh pr list -R NVIDIA/garak --state open --search "aggregate_reports in:body" and "payload_init in:body" → nothing.
• Dedup at filing (2026-09-18, file-level index of the 189 open PRs then): no open PR touched garak/analyze/aggregate_reports.py or tests/analyze/test_aggregate.py.
• Updated 2026-09-19: #2212 is the fix PR for this issue and remains the only open PR touching either file.
• Search across all states for payload_init returns only #930 (the change that introduced the row); for aggregate_reports it returns #2207/#2158/#1783/#1569/#1370 and issues #1610/#1470/#1281/#1368, all of which are about other entry types or the HTML/report_avid path, not payload rows being filtered out.
• Nothing assigned, and no other open issue states this symptom. For orientation, since this is now the third garak/analyze/ item from me: #2213 / PR #2214 are about report_digest.py scoring each probe/detector pairing from only one contributing run — different file, different root cause, no overlap with this one.

Environment

1. macOS (Darwin 27.0.0, arm64)
2. Python 3.11.15
3. direct repository checkout at 8d1259ef
4. garak version reported as 0.17.1.pre1
5. Command-line flags exactly as in the reproduction above; report.jsonl, the aggregate and both digest outputs from that run
6. No special hardware; the reproduction needs no network

I have the two-entry whitelist change plus those three tests ready as a PR.