In a well-known family of LLM evaluation bugs, the failure doesn't look like a failure.
A judge model goes down, and the benchmark keeps publishing scores — because somewhere in the code, a handler like this turned an outage into a legitimate-looking zero: I hit this pattern repeatedly while fixing correctness bugs across mainstream AI/ML open source.
So I did what you do when the same defect family keeps appearing: I built a detector for it — failroute, a static analyzer that flags failures converted into success-looking values.
It's on PyPI, it runs in CI, and every number in this post is reproducible from the repository.
But the interesting finding of the last week is not my tool.
It's about the ecosystem it lives in.
The idiom that learned to hide Python has two ways to write "ignore failures here".
The old one: Every linter has an opinion about that shape.
Ruff's flags try-except-pass.
Bandit's does the same.
The message is always some variant of "consider logging the exception".
The modern one: Semantically identical — the failure is discarded either way.
And here is the part that surprised me: no shipped linter flags it.
Not ruff, not Bandit, not bugbear, not pylint.
The only related rule is bugbear's , which fires on with no arguments — the case where nothing is even suppressed.
It gets better.
Ruff's rule exists specifically to recommend this rewrite: "Use instead of --." A project that runs autofix on converts every flagged handler into a suppress block — and moves all of them out of every existing detector's view in one automated sweep.
The silence is unchanged.
The syntax learned to hide.
I want to be careful about the claim here, because "linter X doesn't catch Y" is easy to assert and hard to defend.
This one is defensible: with valid arguments is a call expression, not an , and every exception-handling rule in the shipped linters dispatches on handlers.
As of today, the detection gap is real, shipped, and — in 's case — actively widened by autofix.
Does it matter in real code? is everywhere in modern Python.
In the source packages of eight real AI/eval repositories (garak, inspect_ai, pydantic-ai, uqlm, trl, smolagents, deepteam, fickling), I counted 77 suppress blocks, sitting alongside 403 silent-fallback / masked-exception handlers that syntactic rules also cannot express — 613 findings total, against 67 for ruff's exception rules.
Each finding in the benchmark is a routing decision a reviewer should have made explicitly: is discarding this failure correct here?
Sometimes yes — that's what the opt-out marker is for.
Often it's the judge-outage pattern wearing a cleaner syntax.
None of this means suppress is bad. is idiomatic cancellation absorption and failroute doesn't flag it, for exactly the reason handler rules exempt .
The point is narrower: a routing decision that used to be visible for review is migrating into a form where review tooling cannot see it, and the migration is automated.
What a detector for this needs (and what I learned building one) Syntactic detection of suppress is trivial — it's one AST pattern.
The hard part is the same as for any semantic defect family: precision, because a linter that cries wolf gets disabled. failroute's answer is a hand-labelled corpus.
Every rule must match fixtures whose ground truth was written from the semantics of each case, independently of tool output, with precision and recall pinned at 1.0.
The corpus has already paid for itself twice: The suppress fixtures were added before the detector existed.
The benchmark ran red — exactly the five expected misses — which is what makes "labels independent of tool output" a testable claim rather than a slogan.
The corpus then caught a design mistake before release: the first draft flagged .
But absorbing cancellation is idiomatic control flow, and the handler rules already exempt .
Same semantics, two syntaxes, two answers — a bug. v0.5.1 unified the ignore lists across both syntaxes.
The broader lesson for anyone building tooling with AI assistance: the AI is good at proposing detector logic and finding the gap; the deterministic gate (a corpus that must first fail, then pass) is what turns a plausible-sounding rule into a defensible one.
The full loop, including the failure cases, is documented in the repository's process doc.
Try it There's a pre-commit hook, a GitHub Action that uploads findings to code scanning, and a self-scan of the repository itself in CI (expected alerts: zero).
If you maintain an eval or red-team framework, I'd genuinely like to know whether it finds anything real in your codebase — and if it finds something that isn't real, I want to hear about that too, because that's the corpus growing.