Training hangs indefinitely after rollout phase completes on 8 B200 GPUs with CP=4, TP=2

Author: euclidgameCreated Jan 23, 2026Updated Sep 6, 2026

Description

Training gets stuck indefinitely after the rollout phase completes and the training phase starts. The system shows only SGLang health check pings for hours with no training progress.

Environment

Hardware: 8× NVIDIA B200 GPUs Container: Slime official container Model: Qwen3-4B

Configuration

# Parallelism settings
--tensor-model-parallel-size 2
--sequence-parallel
--pipeline-model-parallel-size 1
--context-parallel-size 4
--expert-model-parallel-size 1
--expert-tensor-parallel-size 1

# Rollout settings
--rollout-batch-size 16
--n-samples-per-prompt 64
--rollout-max-response-len 40000
--num-steps-per-rollout 1

# Training settings
--colocate
--use-dynamic-batch-size
--max-tokens-per-gpu 32768
--calculate-per-token-loss
--use-slime-router

# SGLang settings
--rollout-num-gpus-per-engine 1
--sglang-mem-fraction-static 0.7

Expected Behavior

Training should proceed after rollout phase completes, showing training metrics and gradient updates.

Actual Behavior

The rollout phase completes successfully with metrics logged:

[2026-01-23 17:02:54] rollout 16: {'rollout/response_lengths': 18408.32, 'rollout/rewards': -8.03e-09, ...}
[2026-01-23 17:02:54] passrate 16: {'passrate/pass@1': 0.243, 'passrate/pass@2': 0.324, ...}
[2026-01-23 17:02:54] Timer actor_train start

After actor_train start, training hangs indefinitely. Only SGLang health check pings appear in the logs:

[2026-01-23 17:03:10] INFO: 172.17.0.2:43964 - "GET /health HTTP/1.1" 200 OK [repeated 8x across cluster]
[2026-01-23 17:03:40] INFO: 172.17.0.2:53212 - "GET /health HTTP/1.1" 200 OK [repeated 8x across cluster]
... (continues for hours with no training progress)
[2026-01-23 17:28:10] INFO: 172.17.0.2:42272 - "GET /health HTTP/1.1" 200 OK [repeated 8x across cluster]

The system remains in this state indefinitely with no training updates.

Additional Notes

The issue is intermittent - encountered multiple times with different training configurations, but not 100% reproducible
Rollout phase always completes successfully; the hang occurs specifically at the training phase start
The hang appears to be related to the actor_train step, possibly a deadlock in distributed training communication or weight synchronization
Health checks indicate SGLang engines remain alive/healthy during the hang
No OOM issues are observed

Source: THUDM/slime

View original on GitHub View discussion on GitHub