Instead of rising steadily, the reward fluctuates wildly

I used format reward and acc reward. Using the qwen2.5-7b-base model, I noticed that for a given system prompt, the format reward steadily increased, but after a certain number of steps it started to decrease and then started to fluctuate. At the same time, the reward also starts to fluctuate. There's a lot of repetition happening at the same time.

The orange line in the picture is the result under $\beta$=0.001 (Coefficient of KL), and the purple line is the result under $\beta$=0. It can be seen that no-zero $\beta$ still cannot avoid making the reward drop to 0, although it can delay the time when the decrease occurs. I also tried different coefficients for KL, but they all gave similar results instead of steadily increasing.

I have a few questions:

1. What might be the reason why rewards fluctuate so much?
2. How can I fix it?

Any comments would really help me!!