Add Z-Loss Support for LM Pretraining
Author: aflah02Created Jun 18, 2026Updated Jun 18, 2026
Labelsfeature request
Currently Z-Loss is only supported for RM Training. It shouldn't be too hard to also add in support for LM-Pretraining. Creating this issue to track progress on this.
Source: EleutherAI/gpt-neox