ray error for multi-nodes training

Author: RushDonCreated May 7, 2026Updated May 13, 2026

for multi-nodes training， ray==2.55.0 doesn't work。 start the script "examples/scripts/train_vlm_math_hybrid_engine.sh" by

ray job submit --address="http://127.0.0.1:8265"
--runtime-env-json='{"working_dir": "."}'
-- python3 -m openrlhf.cli.train_ppo_ray
--ref.num_nodes 1
--ref.num_gpus_per_node 6
--actor.num_nodes 1
--actor.num_gpus_per_node 6
--vllm.num_engines 4
--vllm.tensor_parallel_size 1
--vllm.gpu_memory_utilization 0.7
....

reporting RuntimeError: Request failed with status code 500: No available agent to submit job, please try again later..

So I downgrade to ray==2.48.0. it works well on single node. but for muli-nodes,

node1: ray start --head --node-ip-address 0.0.0.0 --num-gpus 8 node2: ray start --address 10.29.22.34:6379 --num-gpus 8

node1:

error logs are

(PolicyModelActor pid=662055, ip=10.29.22.32) ) (PolicyModelActor pid=662055, ip=10.29.22.32) (norm): Qwen3_5RMSNorm((0,), eps=1e-06) (PolicyModelActor pid=662055, ip=10.29.22.32) (rotary_emb): Qwen3_5TextRotaryEmbedding() (PolicyModelActor pid=662055, ip=10.29.22.32) ) (PolicyModelActor pid=662055, ip=10.29.22.32) ) (PolicyModelActor pid=662055, ip=10.29.22.32) (lm_head): Linear(in_features=4096, out_features=248320, bias=False) (PolicyModelActor pid=662055, ip=10.29.22.32) ) (PolicyModelActor pid=662055, ip=10.29.22.32) ) Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/tmp/ray/session_2026-05-06_18-45-12_298352_230500/runtime_resources/working_dir_files/_ray_pkg_f6cd9140c9c97dd6/openrlhf/cli/train_ppo_ray.py", line 717, in train(args) File "/tmp/ray/session_2026-05-06_18-45-12_298352_230500/runtime_resources/working_dir_files/_ray_pkg_f6cd9140c9c97dd6/openrlhf/cli/train_ppo_ray.py", line 179, in train ray.get(refs) File "/home/c.t/miniforge3/envs/openrlhf/lib/python3.11/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/c.t/miniforge3/envs/openrlhf/lib/python3.11/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/c.t/miniforge3/envs/openrlhf/lib/python3.11/site-packages/ray/_private/worker.py", line 2858, in get values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/c.t/miniforge3/envs/openrlhf/lib/python3.11/site-packages/ray/_private/worker.py", line 958, in get_objects raise value.as_instanceof_cause() ray.exceptions.RayTaskError(OSError): ray::PolicyModelActor.init_model_from_pretrained() (pid=662055, ip=10.29.22.32, actor_id=7adf5e5dec40b8e0e4b49ec105000000, repr=<openrlhf.trainer.ray.ppo_actor.PolicyModelActor object at 0x7ed68deea0d0>) ^^^^^^^^^ torch.distributed.DistStoreError: Timed out after 301 seconds waiting for clients. 1/5 clients joined.

During handling of the above exception, another exception occurred:

ray::PolicyModelActor.init_model_from_pretrained() (pid=662055, ip=10.29.22.32, actor_id=7adf5e5dec40b8e0e4b49ec105000000, repr=<openrlhf.trainer.ray.ppo_actor.PolicyModelActor object at 0x7ed68deea0d0>) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/ray/session_2026-05-06_18-45-12_298352_230500/runtime_resources/working_dir_files/_ray_pkg_f6cd9140c9c97dd6/openrlhf/trainer/ray/ppo_actor.py", line 570, in init_model_from_pretrained self.trainer = ActorPPOTrainer( ^^^^^^^^^^^^^^^^ File "/tmp/ray/session_2026-05-06_18-45-12_298352_230500/runtime_resources/working_dir_files/_ray_pkg_f6cd9140c9c97dd6/openrlhf/trainer/ray/ppo_actor.py", line 106, in init self._init_vllm_sync_group(backend) File "/tmp/ray/session_2026-05-06_18-45-12_298352_230500/runtime_resources/working_dir_files/_ray_pkg_f6cd9140c9c97dd6/openrlhf/trainer/ray/ppo_actor.py", line 149, in _init_vllm_sync_group self._model_update_group = stateless_init_process_group( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/ray/session_2026-05-06_18-45-12_298352_230500/runtime_resources/working_dir_files/_ray_pkg_f6cd9140c9c97dd6/openrlhf/utils/distributed_util.py", line 24, in stateless_init_process_group pg = StatelessProcessGroup.create(host=master_address, port=master_port, rank=rank, world_size=world_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/c.t/miniforge3/envs/openrlhf/lib/python3.11/site-packages/vllm/distributed/utils.py", line 466, in create store = create_tcp_store( ^^^^^^^^^^^^^^^^^ File "/home/c.t/miniforge3/envs/openrlhf/lib/python3.11/site-packages/vllm/distributed/utils.py", line 162, in create_tcp_store socket.close(listen_fd) OSError: [Errno 9] Bad file descriptor (PolicyModelActor pid=662055, ip=10.29.22.32) [MoE] experts_implementation (resolved): eager [repeated 5x across cluster] (PolicyModelActor pid=662055, ip=10.29.22.32) Setting zero3 leaf: Qwen3_5DecoderLayer [repeated 5x across cluster] (PolicyModelActor pid=662055, ip=10.29.22.32) Setting zero3 leaf: ModuleList [repeated 5x across cluster] (PolicyModelActor pid=662368, ip=10.29.22.32) torch_dtype is deprecated! Use dtype instead! [repeated 5x across cluster]

more infos：

ray status ======== Autoscaler status: 2026-05-07 09:55:45.361621 ======== Node status

Active: 1 node_03225a3ba31fd0e0979a5aebc46e18c5b120d64acbb7b283b00bced6 1 node_f6d2f53419183d87a4c226dbd81b9a7dd05fe1e6c9df32b88ac57c8d Pending: (no pending nodes) Recent failures: (no failures)

Resources

Total Usage: 0.0/256.0 CPU 0.0/16.0 GPU 0B/1.10TiB memory 0B/372.53GiB object_store_memory

Total Constraints: (no request_resources() constraints) Total Demands: (no resource demands)

before died, GPU's status are

node1： Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 359320 C VLLM::EngineCore 31332MiB | | 1 N/A N/A 359306 C VLLM::EngineCore 31332MiB | | 2 N/A N/A 359287 C VLLM::EngineCore 31332MiB | | 3 N/A N/A 359299 C VLLM::EngineCore 31332MiB | | 4 N/A N/A 358893 C ...delActor.init_model_from_pretrained 284MiB | | 5 N/A N/A 359041 C ...delActor.init_model_from_pretrained 284MiB | | 6 N/A N/A 359040 C ...delActor.init_model_from_pretrained 284MiB | | 7 N/A N/A 359042 C ...delActor.init_model_from_pretrained 284MiB

node2： | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 753548 C ray::PolicyModelActor 4292MiB | | 1 N/A N/A 753760 C ...delActor.init_model_from_pretrained 4312MiB | | 2 N/A N/A 753761 C ...delActor.init_model_from_pretrained 4312MiB | | 3 N/A N/A 753762 C ...delActor.init_model_from_pretrained 4308MiB | | 4 N/A N/A 753764 C ...delActor.init_model_from_pretrained 4292MiB | | 5 N/A N/A 753763 C ...delActor.init_model_from_pretrained 4308MiB | | 6 N/A N/A 754368 C ...delActor.init_model_from_pretrained 284MiB | | 7 N/A N/A 754367 C ...delActor.init_model_from_pretrained 284MiB

Source: OpenRLHF/OpenRLHF

View original on GitHub View discussion on GitHub