#844·nanochat

Multiple causal-leakage architectures can collapse NanoChat val_bpb to 0.0018

Author: KaiEurekaCreated Aug 31, 2026Updated Sep 1, 2026

@karpathy We found three independently generated train.py causal-leakage hacks without modifying the evaluator or validation data:

  1. 64-token Q/K/V block summariesval_bpb = 0.024213

    Reproducer: train.py

    It averages Q/K/V across each complete 64-token block and broadcasts the summary back to every position. Early positions therefore receive information from later tokens in the same block.

  2. 16-token recurrent chunk summariesval_bpb = 0.076846

    Reproducer: train.py

    It averages each complete 16-token chunk, runs the chunk means through recurrent states, and broadcasts the current chunk state to all its positions. The recurrence is causal between chunks, but not within the current chunk.

  3. Unconstrained spectral Q/K filteringval_bpb = 0.001832

    Reproducer: train.py

    It multiplies the sequence RFFT by an arbitrary learned complex response. Zero-padding alone does not make the resulting circular convolution one-sided, so wrapped negative-lag components let future tokens affect prefix Q/K values.

The first two are block-local aggregation variants; the third is a frequency-domain circular-convolution leak. In all three cases, future information enters the representation before normal causal attention, so the attention mask cannot remove it.

They were discovered by AutoResearch runs driven by OpenCollab. PR #845 adds an architecture-independent prefix/suffix invariance check to reject these models before BPB evaluation.