Differential Transformer loss spikes while training.

I have trained a 1.3B model using both the Differential Transformer and the standard Transformer. I observed a slight improvement in LLM evaluation scores for the Differential Transformer variant, and its loss was consistently lower. However, when I tried the 7B model comparison, training showed loss spikes, and the loss started increasing after a certain point. I also noticed that the gradient norm increased at the same points where the loss spiked. In terms of implementation details, I only used the differential attention part from this repository; I did not include the SwiGLU layer. Instead, I used a standard FFN layer. @YTianZHU