Originally published on hexisteme notes.
I set a threshold from measurement instead of guessing.
The measurement was clean: zero overlap between the two clusters, a 33x gap between them.
I wrote the numbers into a comment with their sample sizes, feeling good about not having guessed.
It was wrong, because the sample I had labelled "known good" was one of the bad ones.
I've written before about checks that cannot fire — guards whose thresholds were miscalibrated for the scale of their input, so nothing you fed them ever tripped the line.
This is a different animal.
My threshold was calibrated from data.
That's exactly what made it convincing, and it's why the calibration itself is where the bug lived.
The check A video pipeline burns captions onto a rendered preview.
A gate then diffs the burned output against the preview and treats every changed pixel as "text we drew," so it can ask whether our captions intrude into the platform's UI safe area.
That reading only holds if the two files are a pair — if this output was burned from this preview.
Nothing verified that.
The only guard compared the number of sampled frames.
Sampling is time-uniform, so two generations whose durations differ by 0.1s both yield exactly 60 samples.
The guard was structurally incapable of noticing the thing it was nominally there to notice.
Setting the threshold I wanted a statistical backstop: if the whole-frame difference between the two files is too large, they probably aren't a pair, so refuse to render a content verdict at all.
Exactly one episode in the repo had both files sitting on disk.
I used it as my positive control. sample median whole-frame abs diff "correctly paired" episode 19.51 known-mismatched pair 98.65 Threshold: 55.0.
Zero overlap, a 33x gap.
Two clusters, cleanly separated.
Done.
The control was a negative That episode's preview file had an mtime nine hours later than its output — and later than the gate run that had already approved it.
The preview on disk had been re-rendered after the burn.
It was never a pair.
So 19.51 wasn't "what a pair looks like." It was "what a non-pair looks like." I had drawn my line using a negative sample as my positive control, and every statistic downstream inherited that.
Manufacturing a real control I couldn't find a verified pair anywhere, so I made one: ran the burn and recorded a content-hash link between the preview and the output at the moment the output was produced.
Then I measured again.
A genuine pair: 2.94.
On a second episode, 3.37.
Both land exactly on the background re-encode noise figure — 3 to 4 — that the same comment file had documented long before.
The evidence had been sitting in my own repo disagreeing with me the whole time.
The real picture had three bands, not two: band values character verified pair 2.94, 3.37 normal same episode, different burn generation 19.31, 19.51 looks normal, produces false verdicts entirely different content 62.88, 97.68, 98.65 obviously wrong What it cost The dangerous band is the middle one, and it sat below my threshold.
So the backstop passed the exact class of failure it existed to stop.
It had already recorded a "212px safe-area violation" against an episode that was published.
Re-run against the genuine pair, that episode is clean.
The 212px blob covered a quarter of the frame — it was two unrelated regions merged by a comparison of two different pictures, not caption creep.
A defect that never existed, filed against an artifact that was already live.
Band three gets caught by any threshold you pick; you don't need measurement to separate 3 from
98.
The only place a threshold does real work is band two — and I had never measured band two at all.
I drew a line between what I believed was band one and what I knew was band three, and the entire region where the check actually operates fell inside the pass zone.
Re-deriving Log-midpoint of the worst verified pair (3.37) and the best-known bad pair (19.31): , so 8.0.
Symmetric in log space, 2.4x