Originally published on hexisteme notes.
I keep my operating judgment in small, versioned files — one file per decision, each with a status, a list of what it supersedes, and a change history.
It's a lightweight way to stop re-litigating the same argument every few weeks: write the judgment down once, and the next time the situation comes up, read the file instead of re-deriving the answer from scratch.
Most of the time this works exactly as intended — a rule gets written, a rule gets proven wrong, a newer rule replaces it, done.
This is the story of the one time that didn't happen, and why "didn't happen" doesn't look like an error.
It looks like a rule sitting there, marked active, doing precisely what it was told to do.
The gate One of my rules governed how to use a panel of eight simulated reviewers I sometimes convene for open-ended judgment calls — different personas critiquing the same plan from different angles.
The panel's response includes, among other things, an "internal consistency" score: roughly, how much the eight answers agree with each other.
Early on I wrote a gate around that number: if internal consistency comes back under 70%, don't adopt the panel's conclusion on its own.
Low agreement, the reasoning went, meant the panel hadn't converged on anything trustworthy.
The supersession, with receipts The gate lasted exactly as long as it took to hit a counterexample.
I ran the panel on an architecture question and got back a response at 21% internal consistency — a reject, by the gate's own arithmetic.
Except when I actually read the eight answers instead of just their agreement score, three of them were pointing at things I hadn't considered at all: a missing measurement step behind a database-performance claim, a conflation of a plan's stated purpose with its surface description, and one persona's terse near-empty answer that turned out to be a legitimate signal — a comment that the question itself wasn't well-formed for consensus, not a dodge.
A fourth response in the same batch was a straightforward hallucination, a decision flow that didn't exist anywhere in the plan, and got discarded on its own merits, unrelated to the consistency score.
None of those three catches would have survived the 70% gate.
The gate wasn't filtering noise — it was filtering the panel's actual value, which was never agreement in the first place.
It was the angles I hadn't thought of.
I wrote a new rule the same day: consistency isn't a pass/fail threshold, it's a diversity reading.
A low score means the eight answers disagree with each other, and disagreement is exactly the condition under which reading each one individually is worth the most, not the least.
The new rule named "reject any response scoring under some cutoff" as the textbook anti-pattern it was correcting.
A second, unrelated check came back at a reassuring 79% consistency and still hid a real disagreement — no fixed threshold works in either direction.
Three months later About a week after writing the rule that killed the gate, I wrote an unrelated rule about diagnosing app growth after launch — a rule that also happens to call the same reviewer panel.
That rule needed its own checklist for what to discard from a panel response, and one of the four discard conditions I wrote into it was: reject if internal consistency is under 70%.
I didn't copy-paste it consciously.
I reasoned my way back to the same number, because it was the number I remembered as correct, and I wrote it in as a self-contained bullet instead of pointing at the rule that had already overturned it.
I re-derived a fact I had personally already retired, and pasted it, inline, into a different document, as if it were still true.
It sat there, active, for about three months.
Then a full sweep across every rule file caught it — not because anything had broken, but because someone went looking for supersession chains and found that this one hadn't reached where it should have.
The fix was one paragraph