[Tracking] Sharded delta weight sync (delta_sharded): roadmap & known issues

Tracking issue for the sharded delta weight sync (delta_sharded checkpoint-engine backend) merged in #6974, and the follow-up roadmap.

Merged

• #6974 — [ckpt,rollout] feat: sharded delta weight sync over NCCL for disaggregated rollout (merged 2026-07-16, 903d90cc)

  • Each trainer rank byte-diffs its local shard against a pinned-CPU snapshot; only changed (position, value) pairs are gathered to rank 0 and streamed to the rollout side. No rank ever stages a full-model mirror.
  • Shard layout is described declaratively via ShardSpec (DeviceMesh + Placement, verl/workers/engine/spec.py) — shared contract with the nccl-m2n work.
  • Measured steady-state sync vs nccl full broadcast (disaggregated, cross-node, offload=True): 7B 3.8s vs 9.1s (2.4×), 32B 12.5s vs 23.2s (1.9×), 72B 12.0s vs 36.9s (3.1×); 200-step GRPO equivalence run and independent re-measurement in the PR thread.

• #7144 — ShardSpec generalization: BlockPlacement block descriptors (any Shard(k) / multi-dim cuts) + explicit-place channel for hybrid geometries (merged 2026-07-27, b9873ee9)

• #7085 — veomni FSDP+EP shard export: fused expert stacks via a dim-0-separable converter (to_hf_chunk + handler-probed hf_slots), HSDP replica dedup (ShardSpec.contributes, covers newer VeOmni's (ep_replicate, ep_fsdp, ep) expert mesh), ep_size=1 degenerate geometry (merged 2026-07-29, ce195c95). Bit-level idempotence verify sweeps green at 30B (EP8 / EP1 / HSDP) and on VeOmni main.

Next steps

• 1. FSDP + EP: Shard(1) / block-descriptor placement — DONE via #7144 + #7085. Original scope: — generalize derive_placement from the flat-offset Shard(0) fast path to a block descriptor (local_shape, global_offset) (both already provided by compute_local_shape_and_global_offset), with a vectorized local→global index translation and block placement in the dense first sync. Covers veomni-style FSDP+EP (gate_up_proj split on dim0 across EP ranks, local experts fully-sharded on dim1). Discussed in the #6974 review.
• 2. Megatron trainer support (bf16 rollout) — DONE via #7181 (merged 2026-07-30, 48ca9eed) (comm-stub probe over Megatron-Bridge mappings; TP+EP+ETP incl. hybrid-Mamba; bit-level verify sweeps + 235B 8.2–9.7× vs same-shape NCCL). PP is out of #7181's scope — see item 5. — mcore shard export through the same ShardSpec contract: 1-D TP DeviceMesh + Shard(partition_dim) + a pure-permutation to_hf converter closure; the engine's converter profile (boundary-preserving gather → NaN-sentinel rebuild → to_hf → HF-coordinate delta) already ships in #6974. (superseded working notes: https://github.com/ChangyiYang/verl/pull/2)
• 3. Quantized rollout (fp8) — in progress (Megatron bf16+QAT trainer → sglang fp8 rollout, quant-domain delta): trainer-side sharded blockwise fp8 quantization (partial absmax grids + one all_reduce(MAX) per record, bitwise-identical to the rollout-side whole-tensor kernel), then byte-diff in code space — unchanged codes under an unchanged scale ship nothing, and measured per-128×128-block absmax flips are exactly zero across training windows at 7B/30B. Bit-level verify sweeps zero-mismatch with QAT enabled at 7B, 30B-A3B (EP8) and 235B-A22B (PP8×EP8); steady sync vs same-config fp8 NCCL: 7B 3.3–3.7s vs 5.0–5.6s, 235B 62–79s vs 306–310s (4–5×). Both masked-copy loader guards from the known issue below (param-storage interval index; post_load_weights outside the patch) are implemented on this branch. PR to follow.
• 4. Composition with P2P / shard-level weight update (nccl-m2n) — reuse the ShardSpec placement to route each shard's delta point-to-point to the rollout ranks that own it, instead of broadcasting the full delta stream to every worker.
• 5. Megatron pipeline parallelism (PP/VPP) for the delta export — draft PR open: #7223 by @gxlvera (rides the bridge's global parameter directory; placeholder rows as zero-count lockstep entries; WORLD merge group with contributes dedup; row-keyed union slot tables also fixing an ep_size>1 expert misattribution. 235B TP4×PP8×EP4 verify sweep zero-mismatch, steady 2.1–2.4× vs same-shape NCCL).

Known issue

• Masked-copy guards in the SGLang delta loader. The in-place sparse apply patches Tensor.copy_ (scoped to the loader's model.load_weights calls) to skip NaN-masked positions. The slime implementation this design references — THUDM/slime#1806 (and its SGLang patch THUDM/slime#1993) — additionally (a) restricts the masked semantics to destinations whose storage belongs to model.named_parameters()/named_buffers() (a data_ptr interval index; non-param writes pass through untouched), and (b) runs post_load_weights / process_weights_after_loading in the unpatched environment so derived tensors (fp8 scales, MoE biases, MLA w_kc/w_vc) are recomputed normally. Neither guard is present in the merged loader. No impact observed in the shipped scope (bf16, standard dense models — the loaders there are pure slice/permute + copy_), but both guards should be ported before widening the scope (quant, models with post-load weight transforms).

• (怀疑 / suspected) The cupy staging pool used by the delta wire broadcast holds its blocks instead of returning them to CUDA: measured 4.15 GB held after the full seed and ~0.6–0.9 GB after each steady sync (7B TP2 e2e; the engine logs cupy staging pool after ... send: held X GB). The engine now calls free_all_blocks() after every send as a mitigation; a proper review of the staging-buffer lifecycle for the fsdp and veomni paths is still owed.