Multiple causal-leakage architectures can collapse NanoChat val_bpb to 0.0018
@karpathy We found three independently generated train.py causal-leakage hacks without modifying the evaluator or validation data:
64-token Q/K/V block summaries —
val_bpb = 0.024213Reproducer: train.py
It averages Q/K/V across each complete 64-token block and broadcasts the summary back to every position. Early positions therefore receive information from later tokens in the same block.
16-token recurrent chunk summaries —
val_bpb = 0.076846Reproducer: train.py
It averages each complete 16-token chunk, runs the chunk means through recurrent states, and broadcasts the current chunk state to all its positions. The recurrence is causal between chunks, but not within the current chunk.
Unconstrained spectral Q/K filtering —
val_bpb = 0.001832Reproducer: train.py
It multiplies the sequence RFFT by an arbitrary learned complex response. Zero-padding alone does not make the resulting circular convolution one-sided, so wrapped negative-lag components let future tokens affect prefix Q/K values.
The first two are block-local aggregation variants; the third is a frequency-domain circular-convolution leak. In all three cases, future information enters the representation before normal causal attention, so the attention mask cannot remove it.
They were discovered by AutoResearch runs driven by OpenCollab. PR #845 adds an architecture-independent prefix/suffix invariance check to reject these models before BPB evaluation.
Source: karpathy/nanochat