I spent two weeks building a computer vision component to estimate how full a plastic container is from drone imagery.
Translucent white containers, whitish chemical product inside, shot obliquely from a drone during field inspections.
The headline number looked good: mean absolute error of 0.055 on fill fraction, Pearson correlation of 0.97.
Then I audited my own evaluation and found that 38 of my 46 test crops had the same physical container sitting in the training set.
The arithmetic was fine.
The problem was the sentence I had wrapped around it: I was presenting 0.055 as the error on containers the model had never seen before.
What makes this worth writing about is that I had the guardrail in place from day one, and it failed three separate times for three unrelated reasons.
Each one is easy to reproduce in any project that trains on frames extracted from video.
Why grouping matters here at all A drone flies over a site and captures a burst.
In my case, 12 frames over 12 seconds.
The same physical container appears in every frame of that burst, from slightly different angles and distances.
If you shuffle those crops randomly into train and test, you are asking the model to recognize a container it has already memorized.
The metric you get back describes interpolation between frames of one burst.
It says nothing about a container the model has never seen.
This is the most common failure in applied ML and everyone knows about it.
Which is exactly why the next part is worth reading.
The guardrail I wrote on day one My dataset module reads the grouping column from config and does not offer a random option at all: The code path for a random split does not exist.
You cannot pass a flag to get one.
I wrote it that way on purpose, on the first day, before there was any data to split.
I still leaked.
Three times.
Leak 1: the group column held the wrong ID was set to , which is what you want.
Group by physical container.
The problem was upstream.
When the level labels came back from annotation, the column in had been populated with the scene ID, not a container ID.
A scene in my pipeline is a temporal cluster of captures, formed by grouping images taken less than 120 seconds apart.
So the config said , the code faithfully grouped by whatever was in the column named , and the actual grouping was by scene.
Several distinct containers share a scene.
The guardrail did exactly what it was told and the result was still wrong.
Lesson: is a claim about the contents of a column, not a property the code can enforce.
Nothing in my pipeline verified that the column named after physical containers actually distinguished physical containers.
Leak 2: the evaluation scripts built their own split This is the one that stung.
I wrote a fix for leak 1: a module that clusters crops into physical containers by spatial proximity of bounding box centers within a scene.
Then I moved on, believing the partitions were now grouped by container.
When I went back to the code weeks later to correct a report, I checked which modules imported that clustering function.
Exactly one did, and it was the script that renders a visual verification sheet.
The grouping module never entered a partition.
The actual evaluation code was doing this: is the source frame.
So the split separated frames, and since the same container appears in all 12 frames of a burst by construction, separating frames separates nothing.
The alternative cross-validation I also reported grouped by filename prefix, which has its own problem (see leak 3).
Neither partition ever grouped by container.
I had written the safe split path, wired the config, removed the unsafe option, and then my evaluation scripts quietly went around all of it.
Lesson: a safe path is only safe if it is the only path.
My module could not be misused.
My scripts never called it.
Leak 3: my cross-site evaluation was not cross-site Separately from the container problem, I had been reporting a stronger result: train on one site, e