Fix ineffective gradient clipping caused by duplicate arguments and mismatched defaults
Author: aflah02Created Sep 18, 2026Updated Sep 18, 2026
Labelsbug
GPT-NeoX currently exposes both clip_grad and gradient_clipping, although only gradient_clipping is consumed by DeepSpeed.
There are two related problems:
clip_gradis unconditionally overwritten bygradient_clipping, so configuring it has no effect.- NeoX defines
gradient_clippingwith a default of1.0and omits default-valued options from the generated DeepSpeed configuration. DeepSpeed’s actual default is0.0, meaning disabled.
Consequently, the commonly used configuration:
gradient_clipping: 1.0is omitted from the generated DeepSpeed config, causing gradient clipping to remain disabled.
Expected behavior:
gradient_clipping: 1.0should be passed to DeepSpeed and enable clipping.- Omitting
gradient_clippingshould use DeepSpeed’s default of0.0. - There should be only one gradient-clipping configuration option.
Proposed fix:
- Change NeoX’s
gradient_clippingdefault from1.0to0.0. - Remove the unused
clip_gradargument and synchronization logic. - Update the generated argument documentation.
- Add tests verifying that an explicitly configured value of
1.0reaches DeepSpeed.
Source: EleutherAI/gpt-neox