百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
R

rerope

> 编程语言
开源

校正的旋转位置嵌入

397 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

校正的旋转位置嵌入

Rectified Rotary Position Embeddings (ReRoPE)

Using ReRoPE, we can more effectively extend the context length of LLM without the need for fine-tuning.

Blog

  • https://kexue.fm/archives/9706 (Chinese)
  • https://kexue.fm/archives/9708 (Chinese)
  • https://normxu.github.io/Rethinking-Rotary-Position-Embedding-2/ (English by @NormXU)
  • https://normxu.github.io/Rethinking-Rotary-Position-Embedding-3/ (English by @NormXU)

Idea

Results

Calculated the loss on llama2-13b with samples_15k.jsonl:

Method loss
RoPE-4k(original llama2-13b) 1.4967
RoPE-8k(original llama2-13b) 8.8615
NTK-RoPE-4k(not dynamic) 1.6081
NTK-RoPE-8k(not dynamic) 1.5417
NTK-RoPE-16k(not dynamic) 1.5163
ReRoPE-w1024-4k 1.4996
ReRoPE-w1024-8k 1.4267
ReRoPE-w1024-16k 1.4001

ReRoPE's performance at training length (4k) has hardly decreased, and it possesses the ideal property of "longer context, lower loss".

Usage

Dependency: transformers 4.31.0

Run python test.py to test chatting or run python eval_loss.py to calculate loss with llama2.

From here and here, we can see what modifications ReRoPE/Leaky ReRoPE has made compared to the original llama implementation.

Other

Triton Implementation of ReRoPE: https://gist.github.com/chu-tianxiang/4307937fd94b49c75b61a6967716bae9

Cite

@misc{rerope2023,
  title={Rectified Rotary Position Embeddings},
  author={Jianlin Su},
  year={2023},
  howpublished={\\url{https://github.com/bojone/rerope}},
}

Communication

QQ discussion group: 67729435, for WeChat group, please add the robot WeChat ID spaces_ac_cn

Issues· 10 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言