Missing a negative sign on the exponent for RoPE
Author: yijing-sieCreated Sep 10, 2026Updated Sep 10, 2026
- File:
labml_nn/transformers/rope/__init__.py - Line:
95 - Quote: The paper suggests using $\Theta = {\theta_i = 10000^{\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$ for the $\frac{d}{2}$ pairs of features.
I thing it should be: $\Theta = {\theta_i = 10000^{-\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$
Source: labmlai/annotated_deep_learning_paper_implementations