8x V100 GPU 上的 qwen2.5-3b OOM 问题
作者: sqs-ustc创建于 2025年4月22日更新于 2026年3月29日
errormessage:ray.exceptions.OutOfMemoryError: Task was killed due to the node running low on memory. Memory on the node (IP: 10.0.0.11, ID: ) where the task (task ID: , name=main_task, pid=41260, memory used=1.54GB) was running was 632.61GB / 661.38GB (0.956503), which exceeds the memory usage threshold of 0.95. Ray killed this worker (ID: ) because it was the most recently scheduled task; to see more information about memory usage on this node, use `ray logs raylet.out -ip 10.0.0.11`. To see the logs of the worker, use `ray logs worker-*out -ip 10.0.0.11. Top 10 memory users:
内容来源: Jiayi-Pan/TinyZero