Originally published on hexisteme notes.
A predecessor note diagnosed three production features that passed every dedicated unit test and never executed at all, and why a unit test structurally can't see that gap.
That note answered three cases I already knew about, because I'd already tripped over them.
It didn't answer the question that matters once you've found three: how do you find the rest — the ones nobody happened to notice yet?
This is that search: the tool that actually works, what it found across seven projects, and a fourth failure shape that the predecessor note's two fixes don't reach at all, because in that fourth shape the code was never the thing that was broken.
The query, before the argument Before any of the specifics, here is the shape of the query, so you can run something like it against your own tables in under a minute: If that comes back near 100%, this note may simply not apply to your codebase, and that's a real result, not a failure to reproduce it.
Keep that in mind through the rest of this — every finding below is downstream of a query shaped like this one, not downstream of reading code and guessing.
Grep is not the detector My first instinct, the same one the predecessor note's fixes point toward, was to grep for the failure shape — a default value, an unpopulated argument, a call site missing a keyword.
In one afternoon it produced both a false positive and a false negative.
The sharper miss: a literal grep for a write path failed to find an statement that was, in fact, live and doing exactly the writing I was looking for.
Grep matched the shape of the bug I expected walking in, not the shape the code actually had.
Everything that survived scrutiny below came from asking a database a question, not from asking a shell how a string was spelled.
The question that works is: of all the rows that exist, how many have this column filled?
In one of the fleet's coordination databases, an table carries 177 rows.
Seven of them have populated.
That column isn't cosmetic: three gate constants downstream read it directly — a discard threshold at 0.30, an absorb-only threshold at 0.40, an empty-consensus verifiable floor at 0.20 — and all three had effectively been evaluating against absence for the overwhelming majority of rows they ever saw.
A second query on the same fleet found 101 queries routed through the council's ask-path, of which 95 sit with no result ever reinjected back into the caller.
Grep would have needed to already know the name of the bug to find either number.
The count needed nothing but the row and the column.
The unit is the column The clearest evidence that this class of bug lives at the column level, not the feature or file level, sits in one seven-row table.
Every row of had and — all seven, no exceptions.
In those same seven rows, ranged from 2 to 52 and from 0.50 to 0.90, current and moving normally.
Same table, same seven rows: one axis fully alive, one axis dead since inception.
A demotion threshold that only fires past four recorded runs, , had been permanently unreachable, because the run counter it depends on had never once incremented.
That's sharp enough to fool a careful reviewer.
A validator working this exact case surfaced as the candidate to investigate, and it was overruled: the defect belonged to the dead axis, , and the investigator had attached it to the axis that was breathing.
The lesson generalizes past this one table — don't ask "is this feature working," because a feature can be half alive.
Ask, column by column, which ones are actually moving.
The root cause, once isolated to the right column, was six lines: the function that executes a council panel expects its caller to hand it a ledger object, and all three real call sites in the codebase constructed the panel without passing one.
Passing it revived the run counter, and as a side effect revived a demotion check that had been silently inert for the identical reason.
Measured after the fix: zero demotions fir