Weight Initialization
Author: simonaayCreated Jul 3, 2024Updated Jul 3, 2024
I'm not sure if what I am saying is correct, but I do not see a place where we're initializing weights in the 3DViT model. I've been experimenting it for 2 days and noticed that the model is not learning effectively and varying around 50%. I'm not sure if that's the reason behind this behavior but I did notice this gap.
Source: lucidrains/vit-pytorch