百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
R

RMT

> 编程语言
开源

(CVPR2024)RMT: Retentive Networks Meet Vision Transformer

391 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

(CVPR2024)RMT: Retentive Networks Meet Vision Transformer

(CVPR2024)RMT: Retentive Networks Meet Vision Transformers

Congratulations! Our work is accepted by CVPR2024

Arxiv

Abstract

Vision Transformer (ViT) has gained increasing attention in the computer vision community in recent years. However, the core component of ViT, Self-Attention, lacks explicit spatial priors and bears a quadratic computational complexity, thereby constraining the applicability of ViT. To alleviate these issues, we draw inspiration from the recent Retentive Network (RetNet) in the field of NLP, and propose RMT, a strong vision backbone with explicit spatial prior for general purposes. Specifically, we extend the RetNet's temporal decay mechanism to the spatial domain, and propose a spatial decay matrix based on the Manhattan distance to introduce the explicit spatial prior to Self-Attention. Additionally, an attention decomposition form that adeptly adapts to explicit spatial prior is proposed, aiming to reduce the computational burden of modeling global information without disrupting the spatial decay matrix. Based on the spatial decay matrix and the attention decomposition form, we can flexibly integrate explicit spatial prior into the vision backbone with linear complexity. Extensive experiments demonstrate that RMT exhibits exceptional performance across various vision tasks. Specifically, without extra training data, RMT achieves 84.8% and 86.1% top-1 acc on ImageNet-1k with 27M/4.5GFLOPs and 96M/18.2GFLOPs. For downstream tasks, RMT achieves 54.5 box AP and 47.2 mask AP on the COCO detection task, and 52.8 mIoU on the ADE20K semantic segmentation task.

Results

Model Params FLOPs Acc log ckpt
RMT-T 14M 2.5G 82.4% RMT-T RMT-T
RMT-S 27M 4.5G 84.1% RMT-S RMT-S
RMT-S* 27M 4.5G 84.8% RMT-S* RMT-S*
RMT-B 54M 9.7G 85.0% RMT-B RMT-B
RMT-B* 55M 9.7G 85.6% RMT-B* RMT-B*
RMT-L 95M 18.2G 85.5% RMT-L RMT-L
RMT-L* 96M 18.2G 86.1% RMT-L* RMT-L*

Citation

If you use RMT in your research, please consider the following BibTeX entry and giving us a star:

bibtex
@inproceedings{fan2023rmt,
  title={RMT: Retentive Networks Meet Vision Transformers},
  author={Qihang Fan and Huaibo Huang and Mingrui Chen and Hongmin Liu and Ran He},
  booktitle={CVPR},
  year={2024}
}

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言