Rotary embeddings pre-allocate 10x more memory than needed
Author: NJX-njxCreated Mar 7, 2026Updated Sep 9, 2026
Problem
In \GPT.init(), rotary embeddings are pre-computed for 10x the actual sequence length:
\\python self.rotary_seq_len = config.sequence_len * 10 # 20480 positions \\
Since \MAX_SEQ_LEN = 2048\ and the training/eval code never exceeds this length, the 10x factor wastes GPU memory on unused cos/sin buffers (10x larger than necessary in bfloat16).
Proposed Fix
Set \self.rotary_seq_len = config.sequence_len. One-line change that reduces the rotary buffer memory by 90% with zero impact on training or evaluation (both are bounded by \MAX_SEQ_LEN).
I'll submit a PR for this.
Source: karpathy/autoresearch