Localization Distillation for Object Detection (CVPR 2022, TPAMI 2023)
Localization Distillation for Object Detection (CVPR 2022, TPAMI 2023)
Analysis of LD in ZhiHu: 目标检测-定位蒸馏 (LD, CVPR 2022) and 目标检测-定位蒸馏续集——logit蒸馏与feature蒸馏之争
This is the code for our paper:
…
[2022.12.3] Rotated-LD-Jittor is now available.
[2022.4.13] Rotated-LD-mmRotate is now available.
[2021.3.30] LD is officially included in MMDetection V2, many thanks to @jshilong , @Johnson-Wang and @ZwwWayne for helping migrating the code.
LD is the extension of knowledge distillation on localization task, which utilizes the learned bbox distributions to transfer the localization dark knowledge from teacher to student.
LD stably improves over GFocalV1 about ~2.0 AP without adding any computational cost!
Knowledge distillation (KD) has witnessed its powerful capability in learning compact models in object detection. Previous KD methods for object detection mostly focus on imitating deep features within the imitation regions instead of mimicking classification logits due to its inefficiency in distilling localization information. In this paper, by reformulating the knowledge distillation process on localization, we present a novel localization distillation (LD) method which can efficiently transfer the localization knowledge from the teacher to the student. Moreover, we also heuristically introduce the concept of valuable localization region that can aid to selectively distill the semantic and localization knowledge for a certain region. Combining these two new components, for the first time, we show that logit mimicking can outperform feature imitation and localization knowledge distillation is more important and efficient than semantic knowledge for distilling object detectors. Our distillation scheme is simple as well as effective and can be easily applied to different dense object detectors. Experiments show that our LD can boost the AP score of GFocal-ResNet-50 with a single-scale 1x training schedule from 40.1 to 42.1 on the COCO benchmark without any sacrifice on the inference speed.
Please refer to INSTALL.md for installation and dataset preparation. Pytorch=1.7 and cudatoolkits=11 are recommended.
Please see GETTING_STARTED.md for the basic usage of MMDetection.
# assume that you are under the root directory of this project,
# and you have activated your virtual environment if needed.
# and with COCO dataset in 'data/coco/'
./tools/dist_train.sh configs/ld/ld_r50_gflv1_r101_fpn_coco_1x.py 8
lr=(samples_per_gpu * num_gpu) / 16 * 0.01
For 2 GPUs and mini-batch size 6, the relevant portion of the config file would be:
optimizer = dict(type='SGD', lr=0.00375, momentum=0.9, weight_decay=0.0001)
data = dict(
samples_per_gpu=3,
For 8 GPUs and mini-batch size 16, the relevant portion of the config file would be:
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
data = dict(
samples_per_gpu=2,
Do not set your samples_per_gpu larger than 3!
We provide several feature imitation methods, including FitNets fitnet, DeFeat decouple, Fine-Grained finegrain, GI location gibox.
bbox_head=dict(
loss_im=dict(type='IMLoss', loss_weight=2.0),
imitation_method='finegrained' # gibox, finegrain, decouple, fitnet
)
If you find trained model very large, please refer to publish_model.py
python tools/model_converters/publish_model.py your_model.pth your_new_model.pth
CUDA_VISIBLE_DEVICES=0 python3 ./tools/benchmark.py configs/ld/ld_gflv1_r101_r50_fpn_coco_1x.py work_dirs/ld_gflv1_r101_r50_fpn_coco_1x/epoch_24.pth
./tools/dist_test.sh configs/ld/ld_gflv1_r101_r50_fpn_coco_1x.py work_dirs/ld_gflv1_r101_r50_fpn_coco_1x/epoch_24.pth 8 --eval bbox
COCO
LD for Lightweight Detectors
Evaluate on the main distillation region only.
| Teacher | Student | Training schedule | AP (val) | AP50 (val) | AP75 (val) | AP (test-dev) | AP50 (test-dev) | AP75 (test-dev) | AR100 (test-dev) |
|---|---|---|---|---|---|---|---|---|---|
| -- | R-18 | 1x | 35.8 | 53.1 | 38.2 | 36.0 | 53.4 | 38.7 | 55.3 |
| R-101 | R-18 | 1x | 36.5 | 52.9 | 39.3 | 36.8 | 53.5 | 39.9 | 56.6 |
| -- | R-34 | 1x | 38.9 | 56.6 | 42.2 | 39.2 | 56.9 | 42.3 | 58.0 |
| R-101 | R-34 | 1x | 39.8 | 56.6 | 43.1 | 40.0 | 57.1 | 43.5 | 59.3 |
| -- | R-50 | 1x | 40.1 | 58.2 | 43.1 | 40.5 | 58.8 | 43.9 | 59.0 |
| R-101 | R-50 | 1x | 41.1 | 58.7 | 44.9 | 41.2 | 58.8 | 44.7 | 59.8 |
| -- | R-101 | 2x | 44.6 | 62.9 | 48.4 | 45.0 | 63.6 | 48.9 | 62.3 |
| R-101-DCN | R-101 | 2x | 45.4 | 63.1 | 49.5 | 45.6 | 63.7 | 49.8 | 63.3 |
Self-LD
Evaluate on the main distillation region only.
| Teacher | Student | Training schedule | AP (val) | AP50 (val) | AP75 (val) |
|---|---|---|---|---|---|
| -- | R-18 | 1x | 35.8 | 53.1 | 38.2 |
| R-18 | R-18 | 1x | 36.1 | 52.9 | 38.5 |
| -- | R-50 | 1x | 40.1 | 58.2 | 43.1 |
| R-50 | R-50 | 1x | 40.6 | 58.2 | 43.8 |
| -- | X-101-32x4d-DCN | 1x | 46.9 | 65.4 | 51.1 |
| X-101-32x4d-DCN | X-101-32x4d-DCN | 1x | 47.5 | 65.8 | 51.8 |
Logit Mimicking vs. Feature Imitation
Ours = Main KD + Main LD + VLR LD. ''Main'' denotes the main distillation region, ''VLR'' denotes the valuable localization region. Teacher is R-101 and student is R-50.
| Method | Training schedule | AP (val) | AP50 (val) | AP75 (val) | APs (val) | APm (val) | APl (val) |
|---|---|---|---|---|---|---|---|
| -- | 1x | 40.1 | 58.2 | 43.1 | 23.3 | 44.4 | 52.5 |
| FitNets | 1x | 40.7 | 58.6 | 44.0 | 23.7 | 44.4 | 53.2 |
| Inside GT Box | 1x | 40.7 | 58.6 | 44.2 | 23.1 | 44.5 | 53.5 |
| Main Region | 1x | 41.1 | 58.7 | 44.4 | 24.1 | 44.6 | 53.6 |
| Fine-Grained | 1x | 41.1 | 58.8 | 44.8 | 23.3 | 45.4 | 53.1 |
| DeFeat | 1x | 40.8 | 58.6 | 44.2 | 24.3 | 44.6 | 53.7 |
| GI Imitation | 1x | 41.5 | 59.6 | 45.2 | 24.3 | 45.7 | 53.6 |
| Ours | 1x | 42.1 | 60.3 | 45.6 | 24.5 | 46.2 | 54.8 |
./tools/dist_test.sh configs/ld/ld_gflv1_r101_r18_fpn_voc.py work_dirs/ld_gflv1_r101_r18_fpn_voc/epoch_4.pth 8 --eval mAP
PASCAL VOC
LD for Lightweight Detectors
Evaluate on the main distillation region only.
| Teacher | Student | Training Epochs | AP | AP50 | AP75 |
|---|---|---|---|---|---|
| -- | R-18 | 4 | 51.8 | 75.8 | 56.3 |
| R-101 | R-18 | 4 | 53.0 | 75.9 | 57.6 |
| -- | R-50 | 4 | 55.8 | 79.0 | 60.7 |
| R-101 | R-50 | 4 | 56.1 | 78.5 | 61.2 |
| -- | R-34 | 4 | 55.7 | 78.9 | 60.6 |
| R-101-DCN | R-34 | 4 | 56.7 | 78.4 | 62.1 |
| -- | R-101 | 4 | 57.6 | 80.4 | 62.7 |
| R-101-DCN | R-101 | 4 | 58.4 | 80.2 | 63.7 |
This is an example of evaluation results (R-101→R-18).
…
pan.baidu pw: ufc8, teacher R101
pan.baidu pw: 5qra, teacher R101DCN
pan.baidu pw: 1bd3, Main LD R101→R18, box AP = 53.0
pan.baidu pw: thuw, Main LD R101DCN→R34, box AP = 56.5
pan.baidu pw: mp8t, Main LD R101DCN→R101, box AP = 58.4
GoogleDrive Main LD + VLR LD + VLR KD R101→R18, box AP = 54.0
GoogleDrive Main LD + VLR LD + VLR KD + GI imitation R101→R18, box AP = 54.4
pan.baidu pw: hj8d, Main LD R101→R18 1x, box AP = 36.5
pan.baidu pw: bvzz, Main LD R101→R50 1x, box AP = 41.1
GoogleDrive Main KD + Main LD + VLR LD R101→R18 1x, box AP = 37.5
GoogleDrive Main KD + Main LD + VLR LD R101→R34 1x, box AP = 41.0
GoogleDrive Main KD + Main LD + VLR LD R101→R50 1x, box AP = 42.1
GoogleDrive Main KD + Main LD + VLR LD + GI imitation R101→R50, box AP = 42.4
GoogleDrive Main KD + Main LD + VLR LD R101→R50 1x, box AP = 42.7
No open issues yet, or sync has not completed.