Rotary embeddings pre-allocate 10x more memory than needed

Author: NJX-njxCreated Mar 7, 2026Updated Sep 9, 2026

Problem

In \GPT.init(), rotary embeddings are pre-computed for 10x the actual sequence length:

\\python self.rotary_seq_len = config.sequence_len * 10 # 20480 positions \\

Since \MAX_SEQ_LEN = 2048\ and the training/eval code never exceeds this length, the 10x factor wastes GPU memory on unused cos/sin buffers (10x larger than necessary in bfloat16).

Proposed Fix

Set \self.rotary_seq_len = config.sequence_len. One-line change that reduces the rotary buffer memory by 90% with zero impact on training or evaluation (both are bounded by \MAX_SEQ_LEN).

I'll submit a PR for this.