A800通信问题。两张卡，一张rollout启动vllm，一张训练，连接不到vllm，是不是通信问题？

Checklist / 检查清单

• I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues，确认这是一个新的问题与讨论。

Question Description / 问题描述

我的rollout脚本： CUDA_VISIBLE_DEVICES=1 NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME=lo
swift rollout
--model /data/ms-swift-main/output_box/v0-20260325-102319/checkpoint-1830
--vllm_data_parallel_size 1
--vllm_gpu_memory_utilization 0.9
--model_type qwen3_vl
--vllm_max_model_len 8192 我的训练脚本： CUDA_VISIBLE_DEVICES=1 NCCL_P2P_DISABLE=1 NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME=lo
export WANDB_API_KEY=your_wandb_api_key export PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' export MAX_PIXELS=602112 export USE_FLASH_ATTENTION=1 torchrun
--nproc_per_node=1
--master_port=29501
-m swift.cli.rlhf
--rlhf_type grpo
--model /data/ms-swift-main/output_box/v0-20260325-102319/checkpoint-1830
--external_plugins /data/ms-swift-main/examples/train/grpo/plugin/plugin_iou.py
--system '/data/ms-swift-main/examples/train/grpo/plugin/prompt.txt'
--output_dir /data/ms-swift-main/output_rl
--dataset '/data/ms-swift-main/dataset/sft_0130/53_rl.json'
--reward_funcs external_box_iou_reward
--train_type full
--freeze_vit False
--torch_dtype bfloat16
--learning_rate 1e-6
--max_completion_length 2048
--num_train_epochs 2
--num_iterations 2
--per_device_train_batch_size 4
--per_device_eval_batch_size 4
--gradient_accumulation_steps 4
--num_generations 4
--async_generate true
--beta 0.001
--warmup_ratio 0.01
--temperature 1.0
--deepspeed zero2
--save_strategy 'steps'
--eval_strategy 'steps'
--eval_steps 1000
--save_steps 160
--save_total_limit 2
--dataloader_num_workers 2
--load_from_cache_file true
--logging_steps 1
--log_completions true
--report_to tensorboard swanlab
--use_vllm true
--vllm_mode server
--vllm_server_host 127.0.0.1
--vllm_server_port 8000
# --target_modules all-linear
# --lora_rank 64
# --lora_alpha 128
我的报错： INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 127.0.0.1:34576 - "GET /health/ HTTP/1.1" 200 OK INFO: 127.0.0.1:34582 - "POST /close_communicator/ HTTP/1.1" 200 OK INFO: 127.0.0.1:34582 - "GET /get_world_size/ HTTP/1.1" 200 OK INFO: 127.0.0.1:34582 - "POST /init_communicator/ HTTP/1.1" 200 OK (EngineCore_DP0 pid=884) INFO 04-09 10:41:31 [init.py:1384] Found nccl from library libnccl.so.2 (EngineCore_DP0 pid=884) INFO 04-09 10:41:31 [pynccl.py:103] vLLM is using nccl==2.27.3 (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] Invocation of collective_rpc method failed (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] Traceback (most recent call last): (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] File "/usr/local/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 777, in _handle_client_request (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] result = method(*self._convert_msgspec_args(method, args)) (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] File "/usr/local/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 416, in collective_rpc (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] return self.model_executor.collective_rpc(method, timeout, args, (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] File "/usr/local/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] return [run_method(self.driver_worker, method, args, kwargs)] (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] File "/usr/local/lib/python3.11/site-packages/vllm/utils/init.py", line 3122, in run_method (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] return func(*args, **kwargs) (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] File "/usr/local/lib/python3.11/site-packages/trl/scripts/vllm_serve.py", line 124, in init_communicator (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] self.pynccl_comm = PyNcclCommunicator(pg, device=self.device) (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] File "/usr/local/lib/python3.11/site-packages/vllm/distributed/device_communicators/pynccl.py", line 139, in init (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] self.all_reduce(data) (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] File "/usr/local/lib/python3.11/site-packages/vllm/distributed/device_communicators/pynccl.py", line 162, in all_reduce (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] self.nccl.ncclAllReduce(buffer_type(in_tensor.data_ptr()), (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] File "/usr/local/lib/python3.11/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 337, in ncclAllReduce (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] self.NCCL_CHECK(self._funcs["ncclAllReduce"](sendbuff, recvbuff, count, (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] File "/usr/local/lib/python3.11/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 291, in NCCL_CHECK (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] raise RuntimeError(f"NCCL error: {error_str}") (EngineCore_DP0 pid=884) ERROR 04-09 10:41:31 [core.py:780] RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)