A small convolutional network beats a plain flatten-and-feed-it-forward network by 7.0 points on CIFAR-10.
That's convolutions, pooling, normalisation and skip connections doing honest work.
Then I shuffled the rows of every image, destroying no information at all, and that 7.0-point margin fell to 0.3.
Same architecture.
Same data, in a strict sense I'll defend in a moment.
Almost the entire advantage, gone.
The experiment Take one fixed permutation of the 32 row indices.
Apply it to every image in the training set and every image in the test set — the same permutation, every time.
That's the whole intervention.
Then train two models twice each — once on natural images, once on shuffled ones: Model Params Natural rows Shuffled rows Flatten → 512 → 10 (MLP) 1,578,506 51.4% 51.7% Small CNN 94,538 58.4% 52.0% CNN's margin +7.0 pts +0.3 pts The baseline is a real fully-connected network, not a single linear layer — .
It has the capacity to learn anything the CNN can; what it lacks is any reason to look at pixels near each other.
Two things in that table are worth sitting with.
The CNN wins the natural case with sixteen times fewer parameters — that's the prior paying for itself.
And in the shuffled case it doesn't just lose its lead; it drops 6.4 points in absolute terms, down to roughly where the linear model already was. "You destroyed the data" — no, and this is the important part This is the objection everyone raises, so let's take it seriously, because the experiment is worthless if the objection holds.
A fixed permutation is a bijection.
Nothing is added, nothing is removed, nothing is averaged or blurred: Every pixel is still there, with its original value.
The transformation is invertible, so no information has been lost in any sense that information theory would recognise.
What changed is which pixels sit next to which.
And here's the control experiment that settles it: the fully-connected model scores the same either way — 51.4% natural, 51.7% shuffled, a difference well inside run-to-run noise.
If the shuffle had damaged the data, the linear model would have suffered too.
It didn't, because it never used the layout in the first place — after , position 400 is just position 400, and a fixed permutation of the input columns is something the first layer absorbs by permuting its own weights.
The network is free to relearn the identical function; only the column labels moved.
So the shuffle removes exactly one thing: the usefulness of the assumption that neighbouring pixels are related.
And that assumption turns out to be worth 6.7 of the CNN's 7.0 points.
Which is the uncomfortable version of the finding: the convolutions, the pooling, the normalisation — all of it — were converting one true fact about photographs into 7 points of accuracy.
Take the fact away and the machinery has nothing left to convert.
What a convolution actually assumes Written out, a conv layer makes three claims about your data, none of which are claims about tensors: Locality.
A 3×3 kernel only ever sees a 3×3 neighbourhood.
This is a bet that meaningful patterns are local — that to recognise an edge you need nearby pixels and not distant ones.
True of photographs.
Not true of a shuffled photograph, where the pixels that formed an edge are now scattered across the image.
Weight sharing.
The same kernel slides over every position, so a feature detected in the top-left uses identical weights to the same feature bottom-right.
This is a bet that what a pattern is doesn't depend on where it is.
It's also where the parameter savings come from — and it's why the shuffled case doesn't merely lose the advantage but wastes capacity, since the kernel is now sharing weights across positions that have nothing in common.
Hierarchy.
Stacked layers assume small patterns compose into larger ones: edges into corners, corners into shapes.
A shuffled image has edges nowhere, so there is nothing to compose.
All three are statements about the world, not the maths.
When t