FocalNet

Focal Modulation Networks

This is the official Pytorch implementation of FocalNets:

"Focal Modulation Networks" by Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan and Jianfeng Gao.

News

• [11/07/2023] Researchers showed that Focal-UNet beats previous methods on several earth system analysis benchmarks. Check out their code, paper, and project!
• [06/30/2023] :collision: Please find FocalNet-DINO checkpoints from huggingface. The old links are deprecated.
• [04/26/2023] By combining with FocalNet-Huge backbone, Focal-Stable-DINO achieves 64.8 AP on COCO test-dev without any test time augmentation! Check our Technical Report for more details!
• [02/13/2023] FocalNet has been integrated to Keras, check out the tutorial!
• [01/18/2023] Checkout a curated paper list which introduce networks beyond attention based on modern convolution and modulation!
• [01/01/2023] Researchers showed that Focal-UNet beats Swin-UNet on several medical image segmentation benchmarks. Check out their code and paper, and happy new year!
• [12/16/2022] :collision: We are pleased to release our FocalNet-Large-DINO checkpoint pretrained on Object365 and finetuned on COCO, which help to get 63.5 mAP without tta on COCO minival! Check it out!
• [11/14/2022] We created a new repo FocalNet-DINO to hold the code to reproduce the object detection performance with DINO. We will be releasing the object detection code and checkpoints there. Stay tunned!
• [11/13/2022] :collision: We release our large, xlarge and huge models pretrained on ImageNet-22K, including the one we used to achieve the SoTA on COCO object detection leaderboard!
• [11/02/2022] We wrote a blog post to introduce the insights and techniques behind our FocalNets in a plain way, check it out!
• [10/31/2022] :collision: We achieved new SoTA with 64.2 box mAP on COCO minival and 64.3 64.4 box mAP on COCO test-dev based on the powerful OD method DINO! We used huge model size (700M) beating much larger attention-based models like SwinV2-G and BEIT-3. Checkoout our new version and stay tuned!
• [09/20/2022] Our FocalNet has been accepted by NeurIPS 2022!
• [04/02/2022] Create a gradio demo in huggingface space to visualize the modulation mechanism. Check it out!

Introduction

We propose FocalNets: Focal Modulation Networks, an attention-free architecture that achieves superior performance than SoTA self-attention (SA) methods across various vision benchmarks. SA is an first interaction, last aggregation (FILA) process as shown above. Our Focal Modulation inverts the process by first aggregating, last interaction (FALI). This inversion brings several merits:

• Translation-Invariance: It is performed for each target token with the context centered around it.
• Explicit input-dependency: The modulator is computed by aggregating the short- and long-rage context from the input and then applied to the target token.
• Spatial- and channel-specific: It first aggregates the context spatial-wise and then channel-wise, followed by an element-wise modulation.
• Decoupled feature granularity: Query token preserves the invidual information at finest level, while coarser context is extracted surrounding it. They two are decoupled but connected through the modulation operation.
• Easy to implement: We can implement both context aggregation and interaction in a very simple and light-weight way. It does not need softmax, multiple attention heads, feature map rolling or unfolding, etc.

Before getting started, see what our FocalNets have learned to perceive images and where to modulate!

Finally, FocalNets are built with convolutional and linear layers, but goes beyond by proposing a new modulation mechanism that is simple, generic, effective and efficient. We hereby recommend:

Focal-Modulation May be What We Need for Visual Modeling!

Getting Started

• Please follow get_started_for_image_classification to get started for image classification.
• Please follow get_started_for_object_detection to get started for object detection.
• Please follow get_started_for_semantic_segmentation to get started for semantic segmentation.

Benchmarking

Image Classification on ImageNet-1K

• Strict comparison with multi-scale Swin and Focal Transformers:

• Strict comparison with isotropic ViT models:

ImageNet-22K Pretraining

NOTE: We reorder the class names in imagenet-22k so that we can directly use the first 1k logits for evaluating on imagenet-1k. We remind that the 851th class (label=850) in imagenet-1k is missed in imagenet-22k. Please refer to this labelmap. More discussion found in this issue.

Object Detection on COCO

• Mask R-CNN