#1432·gpt-neox

Fix ineffective gradient clipping caused by duplicate arguments and mismatched defaults

Author: aflah02Created Sep 18, 2026Updated Sep 18, 2026
Labelsbug

GPT-NeoX currently exposes both clip_grad and gradient_clipping, although only gradient_clipping is consumed by DeepSpeed.

There are two related problems:

  1. clip_grad is unconditionally overwritten by gradient_clipping, so configuring it has no effect.
  2. NeoX defines gradient_clipping with a default of 1.0 and omits default-valued options from the generated DeepSpeed configuration. DeepSpeed’s actual default is 0.0, meaning disabled.

Consequently, the commonly used configuration:

yaml
gradient_clipping: 1.0

is omitted from the generated DeepSpeed config, causing gradient clipping to remain disabled.

Expected behavior:

  • gradient_clipping: 1.0 should be passed to DeepSpeed and enable clipping.
  • Omitting gradient_clipping should use DeepSpeed’s default of 0.0.
  • There should be only one gradient-clipping configuration option.

Proposed fix:

  • Change NeoX’s gradient_clipping default from 1.0 to 0.0.
  • Remove the unused clip_grad argument and synchronization logic.
  • Update the generated argument documentation.
  • Add tests verifying that an explicitly configured value of 1.0 reaches DeepSpeed.