[ROADMAP][2026 Q3] Megatron Core Roadmap

This roadmap outlines the key Megatron Core features, enhancements, and improvements planned for Q3 2026. The 26.08 release has been delivered; remaining items track ongoing Q3 work. This is a tentative roadmap and subject to change.

Last updated: September 15, 2026.

For detailed information on past releases, see the Megatron Core release notes. For the previous quarterly roadmap, see Q2 2026 Roadmap #4997. For component-specific plans and status, see the Q3 MoE Roadmap #6757 and Q3 Megatron FSDP v2 Roadmap #6812.

---

Q3 Roadmap (26.08 delivered; ongoing Q3 work)

MoE

• DeepSeek-V4 architecture support - Integrate CSA/HCA hybrid attention, DSA, and mHC; support Hash MoE and ClampedSwiGLU; add MTP with mHC, fused kernels, Muon/AdamW training recipes, packed sequence handling, and long-context validation (tracker #4468, native hybrid orchestration #6402, Hash MoE #6403, fused mHC #6401; foundational dev PRs #4458, #4481, #4518, and the load-balanced context-parallel DSA indexer #6058).
• Hash MoE and ClampedSwiGLU - Implement Megatron Core support for DeepSeek-V4-specific routing and activation functions (#4481, #5130, #6403).
• MoE communication overlap - ShortcutMoE and THD dispatcher support are delivered (#6959, #5008). ShortcutMoE overlaps MoE all-to-all communication with paired attention, Mamba, or GDN computation; its PR reports 4.7–12.1% higher TFLOP/s and 82–95% overlap efficiency across four evaluated configurations. Latent shared-expert overlap and DeepEP V2 ElasticBuffer work continue (#5135, #4632).
• Optimized MoE kernels - Optimize Blackwell performance by integrating GEMM + SwiGLU fused MLP, mHC, and batchedGEMM fusion techniques.
• MegaMoE / ultra-sparse MoE - Research expanded expert counts, sparsity-driven activations, predictive routing mechanisms, and adaptable recomputation strategies (Megatron-LM #6810, Transformer Engine #3350).
  • MOK implementation - Integrate Mixture-of-Kittens as a pluggable MoE megakernel backend (#6572).

Model Architecture and Support

• DeepSeek-V4 recipes - Complete recipes with end-to-end convergence validation across Megatron-LM and Megatron Bridge (Megatron Bridge DeepSeek-V4 examples).
• Long-context validation - Verify convergence for 4K/16K dense and 64K/1M sparse training setups.
• HybridModel - Introduce a heterogeneous-layer model definition for DeepSeek-V4 and future hybrid configurations, replacing GPTModel (#4537, #5030, #5031, #4538, #4539).
• Gated Delta Product (GDP) — delivered - Add GDP as a first-class HybridModel mixer with training, static and dynamic inference, tensor and context parallelism, packed-sequence support, distributed checkpoint resharding, and optimizer integration (#6074).
• * Ling-V3 Tiny support — delivered on dev - Add training support for the KDA-MLA hybrid sparse-MoE architecture, including packed THD inputs, context parallelism, gated MLA, MTP, and checkpoint-resume validation (#6556).
• Nemotron 3 on HybridModel - Make Nemotron 3 native to HybridModel.
• * DeepSeek-V4 transition to HybridModel - Move DeepSeek-V4 development from GPTModel to the new HybridModel (#5042).
• Qwen3-Next support — delivered - Support the GDN-based hybrid architecture, attention and shared-expert output gates, QK LayerNorm weight-decay handling, context parallelism, and packed sequences (#1989, #2751, #2752, #2753, #2642, #2645).
  • GDN architecture and parallelism — delivered - Support Gated Delta Net for Qwen3-Next, including context parallelism, packed sequences, and unified per-sequence AlltoAll (#1989, #2642, #2645, #4913).
  • Fused GDN attention — WIP - Transformer Engine fused GDN support is available (Transformer Engine #3351); Megatron Core integration is being reapplied in draft #7327, following the revert of #6645.
  • GDN2 — in progress - GDN2 architecture support is available in Megatron Core through Flash Linear Attention (#5765). Optimized GDN2 kernels are available in cuDNN Frontend, and Megatron Core integration is planned soon.
• Qwen3.5 / Qwen3.5-VL support — in progress - Add model architecture logic, packed sequence handling, Qwen3.5-VL validation, and bidirectional checkpoint conversion between Hugging Face and Megatron Core. Foundational dev work has landed (#4748, #4749, #4750, #4751); HybridModel migration and pipeline-parallel integration continue (#6315, #6081).
• SFTDataset class - Add a new pre-tokenized, packed implementation (#5017).

Parallelism

• Megatron FSDP - Continue maturing the FSDP2-compatible API, communication-compute overlap, activation recompute support, and pooled memory allocation. Delivered MFSDP v2 milestones include full-iteration CUDA Graphs, context parallelism, and independently configurable dense/expert hybrid sharding (#7075, #7220, #7012).
• Generalized Tensor Parallelism (GTP) — delivered - Add fine-grained per-weight sharding and on-demand asynchronous all-gather with communication overlap; support CUDA Graphs, low-precision parameter gathers, composition with TP/SP/EP/DDP, Adam and Muon, and distributed checkpointing (#4967).
• NCCL EP support — delivered - Add an expert-parallel transport path and dispatch-manager support (#5129, #4644).
• Next-generation dynamic DP - Support multimodal data and improve performance.
• Communication enhancements / DeepEPv2 - Integrate the DeepEPv2 token dispatcher and enable NCCL EP support.

Performance and Memory

• Deterministic training - Track fixed-environment repeatability, baseline equivalence, restart replay, scale validation, and performance overhead (#5785).
• HybridModel 1F1B overlap — in progress - The common schedule-plan foundation is delivered (#4941); grouped HybridStack support, EP-overlap scheduling, FSDP, and training integration continue (#4942, #4943, #4944).
• Checkpoint load performance - Reduce cross-rank reads for torch_dist load (#4628).
• Reduce MoE peak memory - Avoid full intermediate list/cat buffers and chunk the return all-to-all/unpermute path.
• Full-model CUDA Graph with paged stashing - Advance paged stashing to enable full-iteration CUDA Graph capture and optimize memory for dropless MoE configurations.
• Enhanced attention mechanisms - Advance kernel development for GDN, NSA, and associated Transformer Engine or cuDNN Frontend integrations.
• GDN memory optimization - Implement fine-grained activation offloading and selective recompute strategies for in-projection, conv1d, and gated delta rule logic.

Precision and Optimizers

• MXFP8 and NVFP4 training paths - DeepSeek-V3 NVFP4 training and FP4 execution in the 1F1B expert-parallel all-to-all overlap schedule are delivered (#6841, #6135); continue improving MXFP8/NVFP4 parameter gathering and low-precision GroupedGEMM support.
• Low-precision guide - Document practical low-precision training recipes and optimizer configuration recipes.
• Advanced Muon optimizer features - Implement Newton-Schulz orthogonalization coefficients, FP8 primary weights, and precision-aware behavior for DeepSeek-V4.
• MLA support for Muon - Extend the existing metadata-tagging scheme for splitting MLA up-projections (#5015).
• Model-Optimizer PTQ and QAD - Surface Model-Optimizer-based FP8/NVFP4 post-training quantization (PTQ) and quantization-aware distillation (QAD).
• PTQ and QAD documentation - Surface PTQ and QAD documentation using Nemotron 3 as examples.

Inference

• Sliding-window attention in dynamic batching - Bring dynamic inference to parity with static Transformer Engine attention for SWA models (#5138).
• ETP - Add support for expert tensor parallelism (#4743).
• Async scheduling — delivered - Introduce asynchronous dynamic-inference scheduling and overlap, then enable it by default (#5453, #5549, #5939, #6851).
• Batch-invariant MoE inference — delivered - Add fused vLLM and Transformer Engine backends for gated MoE plus routed block-scale MXFP8 FlashInfer execution with graph-safe bounded decode capacity (#6521, #6864).

Reinforcement Learning

• Durable rollout storage — delivered - Persist completed rollout groups in a checksummed write-through bank with recovery, compaction, and resume integration so completed work survives process interruptions (#6352).
• ReFIT and mutable low-precision weights — delivered - Add NCCL many-to-many model transfer, in-place MXFP8 weight updates that preserve CUDA Graph buffer addresses, and parameter-aligned bounded-memory ReFIT execution (#6712, #6826, #6878).

Multimodal

• MIMO (Multimodal In, Multimodal Out) extensions - Expand the TP/DP heterogeneous primitive and enable autonomous nD parallelism for submodules.
• Pipeline parallel support for the language model - Allow colocated layouts where the destination language grid has PP > 1 while the source encoder grid remains PP = 1 (#4784).
• VLM CUDA Graph support - Capture the language model submodule (#4519).

* : Feature is being developed on the dev branch first and may not be part of the 26.08 main release.

---

How to Provide Feedback

We welcome community input on prioritization. Please:

1. React to items you would like prioritized.
2. Comment on this issue with use cases, constraints, and hardware / model configurations.
3. Open focused feature requests with the enhancement label.
4. Contribute pull requests for roadmap items where possible.

---

Credits

This roadmap reflects the collective efforts of NVIDIA, external contributors, and the Megatron Core community.