Model-graded classify: unparseable judge output silently becomes minimum score

Summary

In the model-graded classification path, a judge response that cannot be parsed to a valid choice is assigned the minimum score in the rubric. Judge malfunction (truncation, rate limits, off-format output, refusals) is thereby converted into the worst possible grade for the model being evaluated — silently, with no error surfaced and no invalid-sample accounting in the reported metrics.

Affected code

Verified against repo HEAD, 2026-08-06:

• evals/elsuite/modelgraded/classify_utils.py:10 — INVALID_STR = "__invalid__"
• evals/elsuite/modelgraded/classify_utils.py:99-101 — on invalid parse: return min(choice_scores.values())
• Choice extraction at :33 (starts_or_endswith) is brittle: any judge output not starting/ending with an exact choice string becomes invalid.

Why this matters

• Correlated, not random: judge parse failures correlate with content (long/complex samples, refusals, specific phrasings), so the penalty does not average out — it systematically taxes particular models and sample classes.
• Invisible: nothing in the aggregated metrics reports how many samples were graded as invalid→minimum. A benchmark consumer cannot tell a judge-malfunction tax from genuine capability difference.
• Measured magnitude: in a 672-call campaign across three production judges, strict-extraction failure rates ranged from 0% to 53% depending on judge. Under this code path, a judge at the high end hands the evaluated model the minimum score on over half the samples, with no warning.

Suggested fix

1. Fail closed to an explicit "invalid" state, excluded from scoring and reported as a first-class count in the metrics.
2. Retry-with-backoff on parse failure before any score assignment.
3. Schema-validated JSON verdicts instead of startswith/endswith string matching.

---

Context: aware the hosted Evals platform sunsets 2026-11-30. A repo-level fix or documented known-issue note would let the many forks and citations inherit the correction.