After fine-tuning, the model performs worse than the pre-trained model.

Thank you very much for providing this project. I fine-tuned the pre-trained model dino_deitsmall8_pretrain.pth on my own dataset (approximately 70,000 samples), but after fine-tuning, the performance is worse than the pre-trained model. Both the visualized attention maps and similarity calculations are inferior to the original dino_deitsmall8 model. This issue persists whether I fine-tuned for 3 epochs or 300 epochs. Interestingly, fine-tuning for 3 epochs yields better results than 300 epochs, which I suspect is due to severe overfitting with longer training. How can I improve the fine-tuning results to surpass the pre-trained model? Below are my training parameters and logs. arch: vit_small batch_size_per_gpu: 20 clip_grad: 3.0 data_path: /data01/wujiacheng/data/mini-imagenet100/train dist_url: env:// drop_path_rate: 0.1 epochs: 300 freeze_last_layer: 1 global_crops_scale: (0.4, 1.0) gpu: 0 local_crops_number: 8 local_crops_scale: (0.05, 0.4) local_rank: 0 lr: 0.0005 min_lr: 1e-06 momentum_teacher: 0.996 norm_last_layer: True num_workers: 10 optimizer: adamw out_dim: 65536 output_dir: ./output patch_size: 8 rank: 0 saveckp_freq: 20 seed: 0 teacher_temp: 0.04 use_bn_in_head: False use_fp16: True warmup_epochs: 10 warmup_teacher_temp: 0.04 warmup_teacher_temp_epochs: 0 weight_decay: 0.04 weight_decay_end: 0.4 world_size: 8 fatal: not a git repository (or any parent up to mount point /) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). Data loaded: there are 72448 images. Student and Teacher are built: they are both vit_small network. Loss, optimizer and schedulers ready. Found checkpoint at ./output/checkpoint.pth => loaded 'student' from checkpoint './output/checkpoint.pth' with msg => loaded 'teacher' from checkpoint './output/checkpoint.pth' with msg => key 'optimizer' not found in checkpoint: './output/checkpoint.pth' => key 'fp16_scaler' not found in checkpoint: './output/checkpoint.pth' => key 'dino_loss' not found in checkpoint: './output/checkpoint.pth' Starting DINO training !

    {"train_loss": 7.3975777105156295, "train_lr": 1.5593881389687975e-05, "train_wd": 0.040003278940542016, "epoch": 0}

{"train_loss": 5.733514768491804, "train_lr": 4.68507966364243e-05, "train_wd": 0.04002299577417647, "epoch": 1} {"train_loss": 5.2813909558200205, "train_lr": 7.810771188315966e-05, "train_wd": 0.04006244911448072, "epoch": 2} {"train_loss": 4.931780147671172, "train_lr": 0.000109364627129896, "train_wd": 0.04012163463495158, "epoch": 3} {"train_loss": 4.612698832708123, "train_lr": 0.00014062154237663252, "train_wd": 0.04020054584522951, "epoch": 4} {"train_loss": 4.365842029254521, "train_lr": 0.00017187845762336835, "train_wd": 0.04029917409181127, "epoch": 5} {"train_loss": 4.147426321377269, "train_lr": 0.00020313537287010427, "train_wd": 0.040417508558998075, "epoch": 6} {"train_loss": 3.9338953712608964, "train_lr": 0.00023439228811683975, "train_wd": 0.040555536270082154, "epoch": 7} {"train_loss": 3.751991960459051, "train_lr": 0.00026564920336357545, "train_wd": 0.040713242088769516, "epoch": 8} {"train_loss": 3.5828048398273182, "train_lr": 0.0002969061186103117, "train_wd": 0.04089060872083998, "epoch": 9} {"train_loss": 3.4191214270823824, "train_lr": 0.0003124969637646053, "train_wd": 0.04108761671604358, "epoch": 10} {"train_loss": 3.2559533344367435, "train_lr": 0.0003124787063868354, "train_wd": 0.04130424447023352, "epoch": 11} {"train_loss": 3.102740913596565, "train_lr": 0.0003124421735549047, "train_wd": 0.04154046822773548, "epoch": 12} {"train_loss": 2.9655698861234483, "train_lr": 0.0003123873695561043, "train_wd": 0.041796262083952464, "epoch": 13} {"train_loss": 2.8421991929675627, "train_lr": 0.000312314300821925, "train_wd": 0.04207159798820551, "epoch": 14} {"train_loss": 2.737220007464685, "train_lr": 0.0003122229759273076, "train_wd": 0.04236644574680994, "epoch": 15} {"train_loss": 2.6325242110915417, "train_lr": 0.00031211340558963266, "train_wd": 0.04268077302638645, "epoch": 16} {"train_loss": 2.543589166044134, "train_lr": 0.0003119856026674665, "train_wd": 0.04301454535740672, "epoch": 17} {"train_loss": 2.453971084536968, "train_lr": 0.00031183958215904727, "train_wd": 0.043367726137973274, "epoch": 18} {"train_loss": 2.389527200201971, "train_lr": 0.00031167536120052953, "train_wd": 0.043740276637833655, "epoch": 19} {"train_loss": 2.319972539567842, "train_lr": 0.0003114929590639708, "train_wd": 0.04413215600262723, "epoch": 20} {"train_loss": 2.270206752373318, "train_lr": 0.0003112923971550707, "train_wd": 0.044543321258365624, "epoch": 21} {"train_loss": 2.2110306938071695, "train_lr": 0.000311073699010659, "train_wd": 0.04497372731614514, "epoch": 22} {"train_loss": 2.167464166599433, "train_lr": 0.000310836890295931, "train_wd": 0.04542332697709117, "epoch": 23} {"train_loss": 2.1112963704771435, "train_lr": 0.0003105819988014412, "train_wd": 0.045892070937534446, "epoch": 24} {"train_loss": 2.0732129593833095, "train_lr": 0.0003103090544398354, "train_wd": 0.04637990779441747, "epoch": 25} {"train_loss": 2.0303954672793636, "train_lr": 0.00031001808924234546, "train_wd": 0.04688678405093154, "epoch": 26} {"train_loss": 1.991264530828437, "train_lr": 0.00030970913735502576, "train_wd": 0.04741264412238291, "epoch": 27} {"train_loss": 1.955514863116947, "train_lr": 0.0003093822350347523, "train_wd": 0.047957430342289026, "epoch": 28} {"train_loss": 1.9239578501082364, "train_lr": 0.0003090374206449606, "train_wd": 0.048521082968701675, "epoch": 29} {"train_loss": 1.8921751010569587, "train_lr": 0.0003086747346511482, "train_wd": 0.049103540190758584, "epoch": 30} {"train_loss": 1.8520111252380684, "train_lr": 0.00030829421961612414, "train_wd": 0.049704738135461564, "epoch": 31} {"train_loss": 1.827290213477295, "train_lr": 0.00030789592019501565, "train_wd": 0.05032461087468126, "epoch": 32} {"train_loss": 1.7935898394645315, "train_lr": 0.0003074798831300259, "train_wd": 0.05096309043238634, "epoch": 33} {"train_loss": 1.76907227727362, "train_lr": 0.00030704615724494824, "train_wd": 0.05162010679209833, "epoch": 34} {"train_loss": 1.7515027310072848, "train_lr": 0.0003065947934394387, "train_wd": 0.052295587904569535, "epoch": 35} {"train_loss": 1.725606787768481, "train_lr": 0.00030612584468304115, "train_wd": 0.05298945969568383, "epoch": 36} {"train_loss": 1.7048891793310115, "train_lr": 0.0003056393660089718, "train_wd": 0.0537016460745804, "epoch": 37} {"train_loss": 1.6839283264810796, "train_lr": 0.00030513541450765866, "train_wd": 0.05443206894199684, "epoch": 38} {"train_loss": 1.6673894567761274, "train_lr": 0.0003046140493200464, "train_wd": 0.05518064819883493, "epoch": 39} {"train_loss": 1.6475110412204423, "train_lr": 0.00030407533163065025, "train_wd": 0.05594730175494325, "epoch": 40} {"train_loss": 1.6299118762588607, "train_lr": 0.00030351932466038016, "train_wd": 0.056731945538120054, "epoch": 41} {"train_loss": 1.6132429480651576, "train_lr": 0.0003029460936591196, "train_wd": 0.057534493503332285, "epoch": 42} {"train_loss": 1.5983621466707076, "train_lr": 0.0003023557058980677, "train_wd": 0.0583548576421517, "epoch": 43} {"train_loss": 1.5876780218255204, "train_lr": 0.00030174823066184505, "train_wd": 0.05919294799240588, "epoch": 44} {"train_loss": 1.5697981989324357, "train_lr": 0.0003011237392403656, "train_wd": 0.060048672648043463, "epoch": 45} {"train_loss": 1.5534752824599236, "train_lr": 0.00030048230492046553, "train_wd": 0.060921937769212856, "epoch": 46} {"train_loss": 1.5501195708767765, "train_lr": 0.0002998240029773068, "train_wd": 0.06181264759255256, "epoch": 47} {"train_loss": 1.5323287861471155, "train_lr": 0.00029914891066554086, "train_wd": 0.06272070444169328, "epoch": 48} {"train_loss": 1.5253395079181784, "train_lr": 0.00029845710721024385, "train_wd": 0.0636460087379685, "epoch": 49} {"train_loss": 1.516570071788686, "train_lr": 0.0002977486737976179, "train_wd": 0.06458845901133499, "epoch": 50} {"train_loss": 1.506209128495075, "train_lr": 0.00029702369356546644, "train_wd": 0.06554795191149967, "epoch": 51} {"train_loss": 1.4922195377108534, "train_lr": 0.0002962822515934326, "train_wd": 0.06652438221925358, "epoch": 52} {"train_loss": 1.48735602203328, "train_lr": 0.00029552443489302044, "train_wd": 0.06751764285801042, "epoch": 53} {"train_loss": 1.4792325724476734, "train_lr": 0.0002947503323973786, "train_wd": 0.0685276249055477, "epoch": 54} {"train_loss": 1.462175175745403, "train_lr": 0.0002939600349508677, "train_wd": 0.06955421760595262, "epoch": 55} {"train_loss": 1.4598413309065905, "train_lr": 0.00029315363529839743, "train_wd": 0.07059730838176684, "epoch": 56} {"train_loss": 1.455925998368622, "train_lr": 0.000292331228074542, "train_wd": 0.07165678284633235, "epoch": 57} {"train_loss": 1.4439415440655652, "train_lr": 0.00029149290979243493, "train_wd": 0.07273252481633499, "epoch": 58} {"train_loss": 1.434533314977204, "train_lr": 0.000290638778832445, "train_wd": 0.07382441632454499, "epoch": 59} {"train_loss": 1.4319354517929321, "train_lr": 0.0002897689354306258, "train_wd": 0.0749323376327542, "epoch": 60} {"train_loss": 1.4208135404418, "train_lr": 0.0002888834816669586, "train_wd": 0.07605616724490595, "epoch": 61} {"train_loss": 1.412368931749885, "train_lr": 0.00028798252145336724, "train_wd": 0.07719578192041875, "epoch": 62} {"train_loss": 1.4076655350576064, "train_lr": 0.0002870661605215287, "train_wd": 0.07835105668770134, "epoch": 63} {"train_loss": 1.3964597912159114, "train_lr": 0.00028613450641045953, "train_wd": 0.07952186485785653, "epoch": 64} {"train_loss": 1.3945600225763246, "train_lr": 0.0002851876684539026, "train_wd": 0.08070807803857477, "epoch": 65} {"train_loss": 1.3920739725462894, "train_lr": 0.0002842257577674886, "train_wd": 0.08190956614821343, "epoch": 66} {"train_loss": 1.3859488471716115, "train_lr": 0.0002832488872357018, "train_wd": 0.08312619743006168, "epoch": 67} {"train_loss": 1.3832804730194992, "train_lr": 0.00028225717149863057, "train_wd": 0.08435783846678938, "epoch": 68} {"train_loss": 1.3744480382337902, "train_lr": 0.0002812507269385135, "train_wd": 0.08560435419507723, "epoch": 69} {"train_loss": 1.37167049871873, "train_lr": 0.0002802296716660824, "train_wd": 0.08686560792042894, "epoch": 70} {"train_loss": 1.3687594512526968, "train_lr": 0.0002791941255066998, "train_wd": 0.08814146133216055, "epoch": 71} {"train_loss": 1.3585772369617382, "train_lr": 0.00027814420998629975, "train_wd": 0.08943177451856754, "epoch": 72} {"train_loss": 1.3593620024786321, "train_lr": 0.00027708004831712355, "train_wd": 0.0907364059822688, "epoch": 73} {"train_loss": 1.3534520602562523, "train_lr": 0.00027600176538326106, "train_wd": 0.09205521265572247, "epoch": 74} {"train_loss": 1.3473127303496661, "train_lr": 0.0002749094877259954, "train_wd": 0.09338804991691528, "epoch": 75} {"train_loss": 1.3404909679830996, "train_lr": 0.00027380334352895234, "train_wd": 0.0947347716052214, "epoch": 76} {"train_loss": 1.3364480797745353, "train_lr": 0.0002726834626030585, "train_wd": 0.09609523003743195, "epoch": 77} {"train_loss": 1.3373393587421922, "train_lr": 0.0002715499763713055, "train_wd": 0.09746927602394875, "epoch": 78} {"train_loss": 1.330167546637554, "train_lr": 0.00027040301785332836, "train_wd": 0.09885675888514518, "epoch": 79} {"train_loss": 1.3304106952639017, "train_lr": 0.000269242721649794, "train_wd": 0.10025752646789025, "epoch": 80} {"train_loss": 1.324928660160013, "train_lr": 0.00026806922392660714, "train_wd": 0.1016714251622331, "epoch": 81} {"train_loss": 1.3187058581022826, "train_lr": 0.00026688266239892756, "train_wd": 0.10309829991824865, "epoch": 82} {"train_loss": 1.3178141796631755, "train_lr": 0.0002656831763150128, "train_wd": 0.10453799426304057, "epoch": 83} {"train_loss": 1.3142834861836998, "train_lr": 0.00026447090643987216, "train_wd": 0.10599035031790024, "epoch": 84} {"train_loss": 1.3069883510387972, "train_lr": 0.00026324599503875094, "train_wd": 0.10745520881561951, "epoch": 85} {"train_loss": 1.3104732318130214, "train_lr": 0.0002620085858604326, "train_wd": 0.10893240911795657, "epoch": 86} {"train_loss": 1.3061490738517152, "train_lr": 0.0002607588241203705, "train_wd": 0.11042178923325201, "epoch": 87} {"train_loss": 1.3041534660614829, "train_lr": 0.00025949685648364666, "train_wd": 0.11192318583419206, "epoch": 88} {"train_loss": 1.3003195396632747, "train_lr": 0.0002582228310477579, "train_wd": 0.11343643427572052, "epoch": 89} {"train_loss": 1.2948374743532158, "train_lr": 0.0002569368973252383, "train_wd": 0.1149613686130927, "epoch": 90} {"train_loss": 1.2913228781086155, "train_lr": 0.00025563920622611143, "train_wd": 0.1164978216200744, "epoch": 91} {"train_loss": 1.2854255201641174, "train_lr": 0.00025432991004018164, "train_wd": 0.11804562480727886, "epoch": 92} {"train_loss": 1.2903377073858164, "train_lr": 0.00025300916241916146, "train_wd": 0.11960460844064469, "epoch": 93} {"train_loss": 1.2885303561765271, "train_lr": 0.00025167711835864, "train_wd": 0.12117460156004825, "epoch": 94} {"train_loss": 1.280728243736435, "train_lr": 0.0002503339341798936, "train_wd": 0.12275543199805172, "epoch": 95} {"train_loss": 1.278711299519096, "train_lr": 0.0002489797675115412, "train_wd": 0.12434692639878328, "epoch": 96} {"train_loss": 1.2754140556913562, "train_lr": 0.0002476147772710459, "train_wd": 0.12594891023694751, "epoch": 97} {"train_loss": 1.2739259430940832, "train_lr": 0.00024623912364606353, "train_wd": 0.1275612078369634, "epoch": 98} {"train_loss": 1.274091598437687, "train_lr": 0.0002448529680756468, "train_wd": 0.12918364239223065, "epoch": 99} {"train_loss": 1.2714416869808878, "train_lr": 0.00024345647323129792, "train_wd": 0.1308160359845169, "epoch": 100} {"train_loss": 1.2743088392359494, "train_lr": 0.00024204980299787825, "train_wd": 0.13245820960346946, "epoch": 101} {"train_loss": 1.265813046703692, "train_lr": 0.00024063312245437699, "train_wd": 0.13410998316624534, "epoch": 102} {"train_loss": 1.2606743107162481, "train_lr": 0.00023920659785453676, "train_wd": 0.13577117553725934, "epoch": 103} {"train_loss": 1.2638890175396098, "train_lr": 0.00023777039660734473, "train_wd": 0.13744160454804835, "epoch": 104} {"train_loss": 1.261428320722704, "train_lr": 0.0002363246872573844, "train_wd": 0.13912108701724746, "epoch": 105} {"train_loss": 1.2582485414959028, "train_lr": 0.0002348696394650582, "train_wd": 0.14080943877067723, "epoch": 106} {"train_loss": 1.2537645654159442, "train_lr": 0.0002334054239866761, "train_wd": 0.14250647466154262, "epoch": 107} {"train_loss": 1.2553655015535454, "train_lr": 0.00023193221265441623, "train_wd": 0.14421200859073383, "epoch": 108} {"train_loss": 1.2516674658168205, "train_lr": 0.00023045017835616045, "train_wd": 0.14592585352723625, "epoch": 109} {"train_loss": 1.2444890861078042, "train_lr": 0.00022895949501520492, "train_wd": 0.14764782152863917, "epoch": 110} {"train_loss": 1.2527960584276414, "train_lr": 0.00022746033756984853, "train_wd": 0.1493777237617461, "epoch": 111} {"train_loss": 1.2495508431827864, "train_lr": 0.00022595288195286493, "train_wd": 0.15111537052328367, "epoch": 112} {"train_loss": 1.2447310118251933, "train_lr": 0.00022443730507085396, "train_wd": 0.1528605712607023, "epoch": 113} {"train_loss": 1.2446837277126919, "train_lr": 0.0002229137847834831, "train_wd": 0.15461313459307524, "epoch": 114} {"train_loss": 1.2448710897212139, "train_lr": 0.00022138249988261222, "train_wd": 0.1563728683320825, "epoch": 115} {"train_loss": 1.2402263224553483, "train_lr": 0.00021984363007131492, "train_wd": 0.15813957950309007, "epoch": 116} {"train_loss": 1.2423123800714986, "train_lr": 0.00021829735594278523, "train_wd": 0.1599130743663086, "epoch": 117} {"train_loss": 1.2355392041360647, "train_lr": 0.00021674385895914847, "train_wd": 0.161693158438041, "epoch": 118} {"train_loss": 1.234681655380842, "train_lr": 0.0002151833214301642, "train_wd": 0.16347963651200906, "epoch": 119} {"train_loss": 1.237066503746056, "train_lr": 0.0002136159264918294, "train_wd": 0.16527231268076012, "epoch": 120} {"train_loss": 1.2333445499148386, "train_lr": 0.00021204185808489087, "train_wd": 0.1670709903571507, "epoch": 121} {"train_loss": 1.2259560344180307, "train_lr": 0.00021046130093325377, "train_wd": 0.16887547229590466, "epoch": 122} {"train_loss": 1.2298583858440408, "train_lr": 0.000208874440522309, "train_wd": 0.17068556061524207, "epoch": 123} {"train_loss": 1.2289611470547661, "train_lr": 0.00020728146307716081, "train_wd": 0.17250105681858127, "epoch": 124} {"train_loss": 1.22763606739453, "train_lr": 0.00020568255554077546, "train_wd": 0.17432176181630496, "epoch": 125} {"train_loss": 1.230241806180406, "train_lr": 0.00020407790555204087, "train_wd": 0.17614747594759253, "epoch": 126} {"train_loss": 1.219652475368858, "train_lr": 0.00020246770142374928, "train_wd": 0.17797799900231495, "epoch": 127} {"train_loss": 1.2224865589719431, "train_lr": 0.00020085213212049332, "train_wd": 0.17981313024299198, "epoch": 128} {"train_loss": 1.2223800104935612, "train_lr": 0.00019923138723649427, "train_wd": 0.1816526684268027, "epoch": 129} {"train_loss": 1.220009788648283, "train_lr": 0.0001976056569733503, "train_wd": 0.18349641182765564, "epoch": 130} {"train_loss": 1.2187102031605566, "train_lr": 0.0001959751321177154, "train_wd": 0.18534415825830974, "epoch": 131} {"train_loss": 1.2208756563558647, "train_lr": 0.00019434000401891157, "train_wd": 0.18719570509254668, "epoch": 132} {"train_loss": 1.2170751430753586, "train_lr": 0.00019270046456647006, "train_wd": 0.18905084928739088, "epoch": 133} {"train_loss": 1.2127445554608, "train_lr": 0.00019105670616761567, "train_wd": 0.1909093874053759, "epoch": 134} {"train_loss": 1.2120006283316596, "train_lr": 0.00018940892172468463, "train_wd": 0.1927711156368536, "epoch": 135} {"train_loss": 1.2069310646883287, "train_lr": 0.00018775730461248642, "train_wd": 0.1946358298223433, "epoch": 136} {"train_loss": 1.206128834070951, "train_lr": 0.0001861020486556131, "train_wd": 0.196503325474921, "epoch": 137} {"train_loss": 1.2064570540446886, "train_lr": 0.00018444334810568909, "train_wd": 0.19837339780264407, "epoch": 138} {"train_loss": 1.210334176716881, "train_lr": 0.00018278139761857936, "train_wd": 0.2002458417310076, "epoch": 139} {"train_loss": 1.2002466429206253, "train_lr": 0.00018111639223154162, "train_wd": 0.20212045192543432, "epoch": 140} {"train_loss": 1.205855048459386, "train_lr": 0.0001794485273403404, "train_wd": 0.20399702281379176, "epoch": 141} {"train_loss": 1.201404860886061, "train_lr": 0.00017777799867631547, "train_wd": 0.20587534860893503, "epoch": 142} {"train_loss": 1.1957137008420133, "train_lr": 0.00017610500228341263, "train_wd": 0.20775522333127328, "epoch": 143} {"train_loss": 1.1990967025337493, "train_lr": 0.00017442973449517609, "train_wd": 0.20963644083135952, "epoch": 144} {"train_loss": 1.200380598763581, "train_lr": 0.00017275239191170895, "train_wd": 0.21151879481249541, "epoch": 145} {"train_loss": 1.1959535594683437, "train_lr": 0.00017107317137659948, "train_wd": 0.21340207885335447, "epoch": 146} {"train_loss": 1.193580395270106, "train_lr": 0.0001693922699538236, "train_wd": 0.21528608643061925, "epoch": 147} {"train_loss": 1.1916153922891326, "train_lr": 0.00016770988490461545, "train_wd": 0.21717061094162746, "epoch": 148} {"train_loss": 1.188758715888304, "train_lr": 0.00016602621366432015, "train_wd": 0.2190554457270306, "epoch": 149} {"train_loss": 1.1935037238754134, "train_lr": 0.00016434145381922282, "train_wd": 0.22094038409345393, "epoch": 150} {"train_loss": 1.185