#1393·gpt-neox

Add Z-Loss Support for LM Pretraining

Author: aflah02Created Jun 18, 2026Updated Jun 18, 2026
Labelsfeature request

Currently Z-Loss is only supported for RM Training. It shouldn't be too hard to also add in support for LM-Pretraining. Creating this issue to track progress on this.