LoRA + colocated vLLM: policy→vLLM weight sync fails end-to-end (dtype, then unmapped adapter names)

Environment

• OpenRLHF v0.11.3 (also present on 0.11.0)
• Ray 2.55.0, vLLM 0.27.1, DeepSpeed 0.19.6, peft 0.20
• Qwen3.5-VL-27B (local checkpoint), 4x RTX 4090, single node
• Config: --train.colocate_all --vllm.enable_sleep --ds.enable_sleep --ds.lora.rank 64 --ds.zero_stage 3 --algo.advantage.estimator group_norm

What works

Full pipeline up to the first weight sync: rollout (vLLM TP4), custom reward_func, 4/4 training steps per episode with sane reward/loss curves. LoRA adapters train fine.

Blocker chain in broadcast_to_vllm (one per layer, peeled in order)

1. dtype mismatch — fixed in #1348. peft upcasts adapters to fp32; ZeRO-3 GatheredParameters preserves that; both sync paths declare/ship param.dtype while the engine side is always bf16 (create_vllm_engines hardcodes dtype: bfloat16):

RuntimeError: Worker failed with error 'mismatch dtype: src torch.float32, dst torch.bfloat16'

2. CUDA-IPC fd handoff requires relaxed yama (environment, not code): after #1348 the sync reaches vLLM and hits

Exception: Worker failed with error 'pidfd_getfd: Operation not permitted'

unless kernel.yama.ptrace_scope=0 (Ubuntu default is 1). Worth a docs note for colocate users.

3. LoRA parameter names have no target in the engine — architectural gap:

ValueError: There is no module or parameter named 'base_model' in Qwen3_5ForConditionalGeneration

named_parameters() on the peft-wrapped model yields base_model.model.* adapter (and base) names; vLLM holds plain HF names. For VLM actors params_to_sync filters to requires_grad, so only adapters are shipped — the engine can never consume them, and the base weights are never refreshed either. There is no adapter merge or name mapping anywhere in the sync path, and no official LoRA RL example script ships with the repo (only LoRA SFT).

Ask

Make LoRA usable in PPO/GRPO with colocated vLLM. Two plausible directions:

1. Merge-then-sync: after each episode, gather base params + apply B@A * alpha/r and sync effective weights under plain HF names (VLM: language model only, matching the frozen-vision-encoder logic already present).
2. vLLM LoRA hot-swap: keep base weights synced once, deliver adapters through vLLM's native LoRA request API.

Happy to test any patch on the setup above. #1348 is a strict prerequisite for either path and is verified to clear the dtype assert on this setup.