#1417·gpt-neox

Upgrade Transformer Engine support from 1.12 to 2.18

Author: aflah02Created Sep 4, 2026Updated Sep 4, 2026
Labelsfeature request

GPT-NeoX currently pins NVIDIA Transformer Engine to transformer-engine[pytorch]==1.12. The latest stable Transformer Engine release is 2.18.0.

This issue tracks upgrading GPT-NeoX to Transformer Engine 2.18 while preserving the existing BF16/FP16 and FP8 Transformer Engine execution paths.

This is not expected to be a dependency-only upgrade. Several Transformer Engine APIs and behaviors used by GPT-NeoX have changed since 1.12 and require compatibility updates.