Use your own pre-trained model for downstream segmentation

I would like to ask you a question, I pre-train on my medical image data set (6k) according to the official pre-training code, and then the downstream task is segmentation, but the fine-tuning results with my own pre-training model is always inferior to the author's pre-training model with imagenet, which is why, is my pre-training data is still too small?