Originally published on hexisteme notes.
I built a quality-control harness to check rendered video deliverables — six axes covering geometry, audio lineage, edit-point clicks, edit-point silence, frame lineage, and segment boundaries.
I ran it against ten synthetic surrogate files with deliberately injected defects.
Thirteen cases, all green.
Then I ran it against actual lecture footage and five cases broke.
The interesting part is not that it broke.
It is that all three root causes were the same shape, and that shape is one I had already been bitten by twice before in the same repository without recognizing it as a category.
A threshold written as an absolute constant is a hidden assumption about the scale of your input.
On content whose scale differs from your fixtures, that check does not fail — it stops firing.
And a check that stops firing reports success, which is why it can live in your suite forever.
Thirteen green fixtures, five real failures The harness is three layers: a pure decision layer with no I/O, an executor that shells out to /, and a verification harness that runs positive and negative controls.
The synthetic surrogates were files I generated with known defects — a 200 ms audio delay, a dropped frame, a shifted segment — so I could assert that each axis fires on the defect it owns and stays quiet on the others.
Every one of those passed.
Then real footage: quiet lecture audio, a static presenter frame, ordinary re-encode noise.
Five failures, three distinct causes.
Cause A — an absolute peak threshold on a click detector The edit-point click detector looked for a discontinuity in the waveform at each junction: Measured on the real file: the peak step at control points — deliberately chosen non-junction locations — had a 95th percentile of 1.43e−2.
The synthetic click I injected to test sensitivity peaked at 2.00e−2.
The injected defect was 1.4x the background and unmistakable in the data, and the threshold of 0.12 swallowed both of them.
On quiet content this detector was structurally incapable of firing.
The fix is not a smaller constant.
It is to stop using a constant: On the same data this separates a clean junction at 0.084 from an injected one at 2.8e+3.
Four orders of magnitude, from a detector that minutes earlier could not distinguish either from zero.
Cause B — an absolute margin on frame comparison The frame-lineage axis has to tell "this output frame came from that source frame" apart from "it came from the one two frames over." I had encoded "distinguishable" as .
When I actually measured it, the signal was 34 to 45 times the re-encode noise.
It was not marginal.
The constant was eating a clean signal.
That last line matters as much as the threshold.
Before asking "did the check pass," ask "on this content, could it have failed?" If the answer is no — a static frame where every candidate correlates at 0.9999 — the honest output is not .
It is .
Cause C — an undocumented applicability limit The third one was not a constant, but it belongs to the same family.
The edit-point silence check works only on silence runs between 90 and 359 ms, and it cannot see a 200 ms delay defect at all.
None of that was written down anywhere.
A check whose applicability limits are unrecorded produces a that nobody can interpret.
Proving the fix tightened rather than loosened Here is the obvious objection to everything above: you changed thresholds until the failures went away.
That objection is correct by default and has to be answered with a measurement, not an assurance.
The measurement is a detection floor: inside the same run, on the same content, inject a synthetic defect at a sweep of amplitudes and record the smallest one that still fires the check. before after Click detector floor (smallest amplitude that fires) 0.05 0.001 — 50x more sensitive Synthetic fixtures still passing 13/13 13/13 Full suite incl. real footage 5 failures 26 cases, 0 failures Fifty times more sensitive with every prior positive control