The training loss is so weird

Description

https://github.com/user-attachments/assets/1c1b31a7-8170-4e99-8abc-21677909809c

https://github.com/user-attachments/assets/9cdc9e36-25e5-45c1-9d5f-dda8f2e3acb1

Steps to Reproduce

1.Training statements：python main.py --text "a hotdog" --workspace trial2 -O2 2.It's the first training session after the previous steps 3.Its training loss is particularly strange, either large in the tens of thousands or zero

Expected Behavior

normal loss not ab normal loss

Environment

pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118