analyze_log and qual_review crash on unscoreable (null) detector scores in attempt records

Description

python -m garak.analyze.analyze_log and python -m garak.analyze.qual_review crash with a TypeError on any report.jsonl whose attempt records contain null detector scores. Detectors return None for outputs they cannot score, and Attempt.as_dict() writes those Nones into the attempt record verbatim. Both analysis CLIs then compare None against a float.

This is the attempt-side twin of the condition #1941 / 7e7c8d8 fixed for eval records — that commit's own message describes it as legitimate: "when a detector returns all-None scores the evaluator writes an eval record with passed == fails == 0 and total_evaluated == 0". The eval-side division is guarded (analyze_log.py:86-89); the attempt-side score normalisation and the qualitative-review bucketing are not.

Steps to reproduce

Offline, using only plugins the project ships, from a checkout at 8d1259ef310e4803cf5a4cc77267fdfdc24434ec (garak's dependencies installed; the repo root on PYTHONPATH if you run from the checkout rather than installing it):

python -m garak --target_type "test.Nones" -p test.Test -d mitigation.MitigationBypass --report_prefix /tmp/nones/run
python -m garak.analyze.analyze_log /tmp/nones/run.report.jsonl       # exit 1
python -m garak.analyze.qual_review -j /tmp/nones/run.report.jsonl    # exit 1

The saved report contains, for every completed attempt:

detector_results = {'mitigation.MitigationBypass': [None, None, None, None, None]}
outputs          = [None, None, None, None, None]

and the matching eval row is passed=0, nones=40, total_evaluated=0 — i.e. the run itself is fine; only the post-run analysis dies.

Current behavior

analyze_log exits 1 and loses everything after the first attempt — no hit-rate lines for the remaining probes, no eval summary, and no completion-rate footer (analyze_log.py:96-102):

  File ".../garak/analyze/analyze_log.py", line 60, in <listcomp>
    normalised_scores = [1 if i >= 0.5 else 0 for i in detector_scores]
TypeError: '>=' not supported between instances of 'NoneType' and 'float'

qual_review exits 1 before producing any review (build_review at line 84; the surrounding try only catches IndexError at line 96):

  File ".../garak/analyze/qual_review.py", line 84, in build_review
    if score > 0.5:
TypeError: '>' not supported between instances of 'NoneType' and 'float'

test.Nones is a test plugin, but the same Nones come from ordinary production paths:

• garak/detectors/base.py:239 — a StringDetector scores None for any output that is None or has text is None; :249 also returns None for an unnormalisable output.
• garak/detectors/base.py:176 — any HFDetector with graceful_fail returns all-None when the model call raises; :191 scores missing outputs None.
• garak/detectors/base.py:285 — a TriggerListDetector on an attempt without notes[triggers] returns [None] * len(outputs).
• Also detectors/leakreplay.py:29, encoding.py:54, snowball.py:27/53, divergence.py:99, propile.py:55, sysprompt_extraction.py:83/125, perspective.py:201, and detectors/always.py:45 (always.Passthru forwards another detector's Nones).

Detector.detect is even typed Iterable[float | None] (detectors/base.py:74), so an analysis consumer must expect None.

Expected behavior

An unscoreable output is neither a hit nor a pass, and must not be counted in the denominator — the same treatment the evaluator already gives it, and what docs/source/reporting.rst:22-24 states: "nones are unscoreable outputs, mirroring the top-level nones and excluded from total_evaluated". So:

• analyze_log.py:60 should normalise only the scored outputs; an all-None detector list yields no hit line, and a mixed list reports its rate over the scored entries.
• qual_review.py:84 should skip unscoreable outputs instead of aborting the review, so they appear in neither failing_examples nor passing_examples.

Counting None as a miss would inflate success and counting it as a pass would deflate it; both contradict the eval rows in the very same file.

Test case

Per contributing guidance, here is the failing case. Adding these two functions to tests/analyze/test_analyze.py and running them on 8d1259ef gives TypeError: '>=' not supported between instances of 'NoneType' and 'float':

def test_analyze_log_unscoreable_detector_scores(tmp_path, capsys):
    """analyze_log must not crash when an attempt reports unscoreable outputs."""
    from garak.analyze.analyze_log import analyze_log

    report_path = tmp_path / "all_unscoreable.report.jsonl"
    records = [
        {"entry_type": "attempt", "status": 1, "uuid": "u1", "probe_classname": "test.Test",
         "prompt": "p", "outputs": [None, None]},
        {"entry_type": "attempt", "status": 2, "uuid": "u1", "probe_classname": "test.Test",
         "prompt": "p", "outputs": [None, None],
         "detector_results": {"mitigation.MitigationBypass": [None, None]}},
        {"entry_type": "eval", "probe": "test.Test", "detector": "mitigation.MitigationBypass",
         "passed": 0, "fails": 0, "nones": 2, "total_evaluated": 0, "total_processed": 2},
    ]
    report_path.write_text("\n".join(json.dumps(r) for r in records) + "\n", encoding="utf-8")

    analyze_log(str(report_path))  # must not raise TypeError

    out = capsys.readouterr().out
    assert "100.00%" not in out, "an unscoreable attempt is not a hit"
    assert "## 1 attempts completed" in out


def test_analyze_log_hit_rate_excludes_unscoreable_outputs(tmp_path, capsys):
    """A mixed attempt reports its hit rate over scored outputs only."""
    from garak.analyze.analyze_log import analyze_log

    report_path = tmp_path / "mixed_unscoreable.report.jsonl"
    records = [
        {"entry_type": "attempt", "status": 2, "uuid": "u1", "probe_classname": "test.Test",
         "prompt": "p", "outputs": ["hit", "miss", None],
         "detector_results": {"mitigation.MitigationBypass": [1.0, 0.0, None]}},
    ]
    report_path.write_text("\n".join(json.dumps(r) for r in records) + "\n", encoding="utf-8")

    analyze_log(str(report_path))

    out = capsys.readouterr().out
    assert "50.00%" in out, "one hit over the two scored outputs"
    assert "33.33%" not in out, "the unscoreable slot must not deflate the rate"

Measured on 8d1259ef: both fail with the TypeError above. tests/analyze/test_qual_review.py has the matching pair, which fails on qual_review.py:84. Nothing in the existing suite covers this — all seven committed report.jsonl fixtures under tests/_assets/ contain zero null attempt scores.

Duplicate check

• gh pr list -R NVIDIA/garak --state open --search "analyze_log in:body" / "qual_review in:body" → nothing.
• Dedup at filing (2026-09-18, file-level scan of all open PRs): no open PR touched garak/analyze/analyze_log.py or garak/analyze/qual_review.py — within garak/analyze/, only #2034 (wilson_ci.py, bootstrap_ci.py, ci_calculator.py, rebuild_cis.py) and #2090 (report_digest.py).
• Updated 2026-09-19: #2210 is the fix PR for this issue and is still the only open PR touching either file.
• #1941 (merged, 7e7c8d8) guards the eval-record division only. #1948 (closed, unmerged) is a zero/missing-total variant. #1954 is open and concerns evaluators/base.py accounting, not the analysis consumers. Nothing assigned.

Environment

1. macOS (Darwin 27.0.0, arm64)
2. Python 3.11.15
3. direct repository checkout at 8d1259ef
4. garak version reported as 0.17.1.pre1
5. Flags as in the repro above; report.jsonl and both tracebacks from that run
6. No hardware-specific requirements; the repro needs no network

I have a branch with the two-line fix and the four tests, ready as a PR — happy to hold it if you'd rather decide the denominator semantics differently first.