Proposal: harness-level failure attribution in eval reports (model vs harness vs eval definition)

When an eval run produces failures, there's currently no way to tell from the results where each failure originated. Recorder.record_error in evals/record.py writes exception type and message as opaque events, and the final report only aggregates metrics — so a run whose harness misfired (broken tool wiring, a scorer expectation that silently no-ops) and a run where the model genuinely can't do the task can look identical. The fixes are completely different (fix the harness vs. revisit the eval definition vs. work on the model), and benchmark conclusions get corrupted when harness faults are silently counted as model failures.

I'd like to propose a small attribution layer:

• classify failure origins at the harness boundary — model / harness / eval-definition — where errors and scoring outcomes are already recorded
• surface per-failure-class counts in the report alongside existing metrics
• keep raw events unchanged, so the classification is additive metadata rather than a change to existing record semantics

Concrete validation: inject a harness fault into an existing elsuite eval (e.g. point a tool call at a dead endpoint) and show the report attributes those samples to harness rather than model, while an untouched run attributes its failures to the model. Injected-fault attribution is the test that would catch the motivating case: in langchain-ai/deepagents#6048, TrajectoryScorer.expect(tool_calls=...) accepted tool-identity expectations but the logging path silently ignored them, so misconfigured scoring was indistinguishable from genuine eval outcomes. Ecdysis (arXiv 2609.11677) makes the same model-vs-harness distinction explicit for agent evaluation.

The natural seam seems to be record_error plus report aggregation in evals/record.py, with the classifier pluggable so individual evals can extend it. Would this fit the direction here? If so I'm happy to draft it as a PR.