Great difficulty reproducing training

Has anyone trained EfficientNet-B0 from scratch on ImageNet and successfully reproduced the results? I used the model in this repo and tried to follow the hyperparameters as closely as I can. I even implemented a modified RMSprop in pytorch that matches its Tensorflow counterpart (there's difference in treatment of epsilon). I used standard preprocessing. My setup is 8 GPUs, each GPU computes a batch of 32 images. The learning rate is properly scaled (0.016).

So far my best effort seems to be quite far below the reported numbers (> 3 points lower in top1 acc). The only difference I can think of is Exponential Model Averaging, which the official Tensorflow repo includes. But I highly doubt that EMA makes such a huge difference.

Are there anything else in the model itself that may change the training dynamics?