Digest of an aggregated report scores each probe/detector pairing from one contributing run only

Steps to reproduce

Offline, from a checkout at 8d1259ef, using only shipped plugins and the built-in test target. One probe run in two chunks, then aggregated the way docs/source/analyze.aggregate_reports.rst describes:

garak -t test -n 2 -p test.Test -d always.Pass -g 8 --report_prefix /tmp/g/r1
garak -t test -n 2 -p test.Test -d always.Pass -g 4 --report_prefix /tmp/g/r2
python -m garak.analyze.aggregate_reports -o /tmp/g/both.report.jsonl /tmp/g/r1.report.jsonl /tmp/g/r2.report.jsonl
python -c "import garak.analyze.report_digest as rd; d = rd.build_digest('/tmp/g/both.report.jsonl'); \
print(d['eval']['test']['test.Test']['always.Pass']); print(d['eval']['test']['test.Test']['_summary']['probe_counts'])"

The two runs write 64/64 and 32/32 (ok on 64/64, ok on 32/32 in the console output above). The aggregated report therefore holds two eval rows and two probe_summary rows for the same test.Test / always.Pass pairing, which is what aggregate_reports is meant to produce.

Expected behavior

The digest describes the whole aggregated run: 96 evaluations for that pairing, a confidence interval covering both chunks (or none, if it cannot be computed jointly), and a probe summary consistent with the detector row.

Current behavior

detector row   : {"absolute_defcon": 5, "absolute_score": 1.0, "passed": 64, "total_evaluated": 64}
probe summary  : {"total_evaluated": 32, "nones": 0} / {"detectors": ["always.Pass"], "passed": 32, "fails": 0, "nones": 0}

Two thirds of the run are missing, and the two halves of the same digest disagree with each other: the detector row reports the 64-attempt chunk while _summary reports the 32-attempt chunk. Neither is 96. The confidence/absolute_confidence_lower/absolute_confidence_upper fields on that row are the interval of one chunk, presented as if they described the aggregated total.

A single row also mixes two different chunks, and the mixture is worse than "one chunk wins". _get_detectors_info selects the pairing's rows order by score asc (report_digest.py:361) and build_digest writes the entry once per row, so the last write — the highest-scoring chunk — sets the score, while the counts come from a separate query with limit 1 (:661), i.e. the first row in the file. Two chunked runs of one pairing, 0/10 and 10/10, were digested on 8d1259ef as:

So a pairing that failed half of a chunked run is reported as minimal risk either way, with counts that depend on which report happened to be passed to aggregate_reports first. A run split this way exits 0 and the digest shows no sign of it.

garak version

0.17.1.pre1, source checkout at 8d1259ef310e4803cf5a4cc77267fdfdc24434ec (git log -1 --pretty=oneline).

Additional Information

1. macOS 27.0 (arm64), Python 3.14.5 — note this is above the CI matrix (3.11–3.13); the code path is plain dict/list handling in _parse_report, with nothing version-specific in it.
2. Install method: repository checkout, PYTHONPATH=$PWD.
3. Root cause: garak/analyze/report_digest.py keeps one entry per pairing. _parse_report appends every eval row it reads, and build_digest/digest_to_tbsa then read that list expecting one row per probe/detector pairing; probe_summary rows were overwritten per probe on a last-write-wins basis (probe_summaries[record["probe"]] = record). Nothing pools the counts.
4. aggregate_reports itself is not wrong — it is documented as the tool for "a report that's been run one probe at a time", and it copies each source report's rows verbatim. The digest is the consumer that has to fold them.
5. Reproduced again in unit form: a report built by appending two eval rows for one pairing to the committed tests/_assets/analyze/test.report.jsonl header fails the same way through build_digest() (assert 8 == 24).
6. Related but distinct: #2211 (aggregation drops payload_init/tree_data rows) is about rows that never reach the aggregate at all; this one is about rows that do reach it and are then folded wrongly by the digest.