Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
R

RMT

> 编程语言
Open source

(CVPR2024)RMT: Retentive Networks Meet Vision Transformer

391 stars0 likes0 views
WebsiteGitHub

About

(CVPR2024)RMT: Retentive Networks Meet Vision Transformer

(CVPR2024)RMT: Retentive Networks Meet Vision Transformers

Congratulations! Our work is accepted by CVPR2024

Arxiv

Abstract

Vision Transformer (ViT) has gained increasing attention in the computer vision community in recent years. However, the core component of ViT, Self-Attention, lacks explicit spatial priors and bears a quadratic computational complexity, thereby constraining the applicability of ViT. To alleviate these issues, we draw inspiration from the recent Retentive Network (RetNet) in the field of NLP, and propose RMT, a strong vision backbone with explicit spatial prior for general purposes. Specifically, we extend the RetNet's temporal decay mechanism to the spatial domain, and propose a spatial decay matrix based on the Manhattan distance to introduce the explicit spatial prior to Self-Attention. Additionally, an attention decomposition form that adeptly adapts to explicit spatial prior is proposed, aiming to reduce the computational burden of modeling global information without disrupting the spatial decay matrix. Based on the spatial decay matrix and the attention decomposition form, we can flexibly integrate explicit spatial prior into the vision backbone with linear complexity. Extensive experiments demonstrate that RMT exhibits exceptional performance across various vision tasks. Specifically, without extra training data, RMT achieves 84.8% and 86.1% top-1 acc on ImageNet-1k with 27M/4.5GFLOPs and 96M/18.2GFLOPs. For downstream tasks, RMT achieves 54.5 box AP and 47.2 mask AP on the COCO detection task, and 52.8 mIoU on the ADE20K semantic segmentation task.

Results

Model Params FLOPs Acc log ckpt
RMT-T 14M 2.5G 82.4% RMT-T RMT-T
RMT-S 27M 4.5G 84.1% RMT-S RMT-S
RMT-S* 27M 4.5G 84.8% RMT-S* RMT-S*
RMT-B 54M 9.7G 85.0% RMT-B RMT-B
RMT-B* 55M 9.7G 85.6% RMT-B* RMT-B*
RMT-L 95M 18.2G 85.5% RMT-L RMT-L
RMT-L* 96M 18.2G 86.1% RMT-L* RMT-L*

Citation

If you use RMT in your research, please consider the following BibTeX entry and giving us a star:

bibtex
@inproceedings{fan2023rmt,
  title={RMT: Retentive Networks Meet Vision Transformers},
  author={Qihang Fan and Huaibo Huang and Mingrui Chen and Hongmin Liu and Ran He},
  booktitle={CVPR},
  year={2024}
}

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Python

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言