[Bug] Resume advances Megatron scheduler with rollout_id instead of train step
Bug Description
On resume, slime does:
iteration, _ = load_checkpoint(...)
opt_param_scheduler.step(increment=iteration * args.global_batch_size)But in slime, the saved Megatron iteration is effectively used as rollout_id, not true train step:
- save path uses save(rollout_id, ...)
- load path uses loaded_rollout_id = iteration
- then start_rollout_id = loaded_rollout_id + 1
This is fine for rollout/data-order resume, but not for scheduler resume.
Steps to Reproduce
opt_param_scheduler is stepped per successful optimizer update:
opt_param_scheduler.step(increment=args.global_batch_size)
A rollout may contain multiple train steps (num_steps_per_rollout > 1), so rollout_id * global_batch_size is not the correct scheduler position.
Also, Megatron checkpoints already save/load scheduler state, so this extra step(...) can double-advance the scheduler.
Expected Behavior
Resume can produce right LR/WD schedule.
Actual Behavior
Resume can produce wrong LR/WD schedule, especially when:
- num_steps_per_rollout > 1
- non-constant LR schedule is used
Current ckpt tests do not catch this because they use constant LR and num_steps_per_rollout == 1.
Environment
- slime version:
- Python version:
- PyTorch version:
- CUDA/ROCm version:
- GPU type and count:
- OS:
- SGLang version (if relevant):
- Megatron-LM version (if relevant):
Logs
Additional Context
No response
Pre-submission Checklist
- I have read the CONTRIBUTING.md and understand the collaboration scope.
- I have read the documentation and my issue is not addressed there.
- I have searched for existing issues and this is not a duplicate.
- I have provided a minimal, reproducible example.
Source: THUDM/slime