Missing a negative sign on the exponent for RoPE

Author: yijing-sieCreated Sep 10, 2026Updated Sep 10, 2026
  • File: labml_nn/transformers/rope/__init__.py
  • Line: 95
  • Quote: The paper suggests using $\Theta = {\theta_i = 10000^{\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$ for the $\frac{d}{2}$ pairs of features.

I thing it should be: $\Theta = {\theta_i = 10000^{-\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$

Source: labmlai/annotated_deep_learning_paper_implementations