NaN / non-finite gradients when training IC-LoRA

I’m training an IC-LoRA / video-to-video LoRA setup and I’m seeing frequent non-finite gradients during training.

The loss itself is still finite, but after backward / grad clipping the trainer reports non-finite gradients and skips the optimizer step.

Model: LTX-2.3 22B
Training mode: LoRA
LoRA rank / alpha: 128 / 128
Precision: bf16
Distributed: FSDP2
Learning rate: 1e-4
max_grad_norm: 1.0

In my logs, the issue seems to happen during backward rather than forward, because the loss remains finite. The skipped steps are also often associated with high sigma values.