[BUG] 稳定训练时出现一半OOM，已开启cp

(SGLangEngine pid=22439) [2025-12-17 14:07:56] INFO: 33.184.122.180:62506 - "POST /generate HTTP/1.1" 200 OK (SGLangEngine pid=22439) [2025-12-17 14:07:56] INFO: 33.184.122.180:60048 - "POST /abort_request HTTP/1.1" 200 OK (SGLangEngine pid=22439) [2025-12-17 14:07:56 TP3] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:56 TP7] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:56 TP2] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:56 TP5] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:56 TP4] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:56 TP6] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:56 TP1] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:56 TP0] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:56] INFO: 33.184.122.180:60056 - "GET /flush_cache HTTP/1.1" 200 OK (SGLangEngine pid=22439) [2025-12-17 14:07:57 TP6] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:57 TP2] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:57 TP1] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:57 TP0] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:57 TP3] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:57 TP4] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:57 TP5] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:57 TP7] Cache flushed successfully! (SGLangEngine pid=22439) [2025-12-17 14:07:58] INFO: 33.184.122.180:60058 - "POST /release_memory_occupation HTTP/1.1" 200 OK (MegatronTrainRayActor pid=32167) [2025-12-17 14:07:58] timer.py:24 - Timer wake_up start (MegatronTrainRayActor pid=32167) [2025-12-17 14:07:58] memory_utils.py:41 - [Rank 0] Memory-Usage before wake_up model: {'gpu': '0', 'total_GB': 95.0, 'free_GB': 87.12, 'used_GB': 7.88, 'allocated_GB': 24.93, 'reserved_GB': 25.73} (MegatronTrainRayActor pid=32442) [2025-12-17 14:08:04] reloadable_process_group.py:152 - Reloading 20 process groups in pid 32442 (MegatronTrainRayActor pid=32437) [2025-12-17 14:07:58] memory_utils.py:41 - [Rank 6] Memory-Usage before wake_up model: {'gpu': '6', 'total_GB': 95.0, 'free_GB': 88.74, 'used_GB': 6.26, 'allocated_GB': 25.51, 'reserved_GB': 26.23} [repeated 7x across cluster] (MegatronTrainRayActor pid=32442) [2025-12-17 14:08:04] memory_utils.py:41 - [Rank 4] Memory-Usage after wake_up model: {'gpu': '4', 'total_GB': 95.0, 'free_GB': 63.42, 'used_GB': 31.59, 'allocated_GB': 25.51, 'reserved_GB': 25.95}