经过微调后,模型的性能比预训练模型差。

Thank you very much for providing this project. I fine-tuned the pre-trained model dino_deitsmall8_pretrain.pth on my own dataset (approximately 70,000 samples), but after fine-tuning, the performance is worse than the pre-trained model. Both the visualized attention maps and similarity calculations are inferior to the original dino_deitsmall8 model. This issue persists whether I fine-tuned for 3 epochs or 300 epochs. Interestingly, fine-tuning for 3 epochs yields better results than 300 epochs, which I suspect is due to severe overfitting with longer training. How can I improve the fine-tuning results to surpass the pre-trained model? Below are my training parameters and logs. arch: vit_small batch_size_per_gpu: 20 clip_grad: 3.0 data_path: /data01/wujiacheng/data/mini-imagenet100/train dist_url: env:// drop_path_rate: 0.1 epochs: 300 freeze_last_layer: 1 global_crops_scale: (0.4, 1.0) gpu: 0 local_crops_number: 8 local_crops_scale: (0.05, 0.4) local_rank: 0 lr: 0.0005 min_lr: 1e-06 momentum_teacher: 0.996 norm_last_layer: True num_workers: 10 optimizer: adamw out_dim: 65536 output_dir: ./output patch_size: 8 rank: 0 saveckp_freq: 20 seed: 0 teacher_temp: 0.04 use_bn_in_head: False use_fp16: True warmup_epochs: 10 warmup_teacher_temp: 0.04 warmup_teacher_temp_epochs: 0 weight_decay: 0.04 weight_decay_end: 0.4 world_size: 8 fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). Data loaded: there are 72448 images. Student and Teacher are built: they are both vit_small network. Loss, optimizer and schedulers ready. Found checkpoint at ./output/checkpoint.pth => loaded 'student' from checkpoint './output/checkpoint.pth' with msg => loaded 'teacher' from checkpoint './output/checkpoint.pth' with msg => key 'optimizer' not found in checkpoint: './output/checkpoint.pth' => key 'fp16_scaler' not found in checkpoint: './output/checkpoint.pth' => key 'dino_loss' not found in checkpoint: './output/checkpoint.pth' Starting DINO training! {"train_loss": 7.3975777105156295, "train_lr": 1.5593881389687975e-05, "train_wd": 0.040003278940542016, "epoch": 0} {"train_loss": 5.733514768491804, "train_lr": 4.68507966364243e-05, "train_wd": 0.04002299577417647, "epoch": 1} {"train_loss": 5.2813909558200205, "train_lr": 7.810771188315966e-05, "train_wd": 0.04006244911448072, "epoch": 2} {"train_loss": 4.931780147671172, "train_lr": 0.000109364627129896, "train_wd": 0.04012163463495158, "epoch": 3} {"train_loss": 4.612698832708123, "train_lr": 0.00014062154237663252, "train_wd": 0.04020054584522951, "epoch": 4} …