[BUG] 共享专家重叠导致独立专家的 wgrad 等待到输入梯度合并

作者: fkuner创建于 2026年9月16日更新于 2026年9月18日
标签community-requestwaiting-on-maintainers

Describe the bug

When moe_shared_expert_overlap=True and overlap_dispatch_backward_with_experts_wgrad=True are enabled together, the Flex dispatcher's autograd graph makes routed-expert weight-gradient (wgrad) submission depend on the shared/routed input-gradient merge. This introduces a dependency on shared-expert backward before wgrad can be submitted, undermining the intended dispatch-backward/wgrad overlap.

内容来源: NVIDIA/Megatron-LM