Upgrade Transformer Engine support from 1.12 to 2.18
Author: aflah02Created Sep 4, 2026Updated Sep 4, 2026
Labelsfeature request
GPT-NeoX currently pins NVIDIA Transformer Engine to transformer-engine[pytorch]==1.12. The latest stable Transformer Engine release is 2.18.0.
This issue tracks upgrading GPT-NeoX to Transformer Engine 2.18 while preserving the existing BF16/FP16 and FP8 Transformer Engine execution paths.
This is not expected to be a dependency-only upgrade. Several Transformer Engine APIs and behaviors used by GPT-NeoX have changed since 1.12 and require compatibility updates.
Source: EleutherAI/gpt-neox