[Roadmap] OpenRLHF on Intel XPU

Feature request

Intel XPU support for OpenRLHF's Ray + vLLM + DeepSpeed training path.

Motivation

OpenRLHF has mature NVIDIA/CUDA support, but Intel XPU support lags behind. This roadmap aims to address Intel XPU's gaps over the next 1-2 quarters for OpenRLHF. This issue will continue to be updated to track items in the longer horizon.

P0 - Accelerator portability test coverage

• Propagate oneCCL log level to Ray workers.

  • PR #1267:

• Run loss aggregation tests on available accelerators.

  • PR #1268:

• Add device-generic distributed backend smoke tests.

  • PR #1269:

P1 - Use device-agnostic PyTorch accelerator APIs

• Replace applicable hard-coded torch.cuda.* calls with device-agnostic torch.accelerator.* APIs.
• Validate the affected training paths on Intel XPU without changing existing CUDA behavior.
• PR #1301:

P2 - Make flash_attn utility imports optional

• Make flash-attn and ring-attention imports optional so OpenRLHF can run on XPU without these CUDA-specific packages.
• Preserve the current behavior when these dependencies are installed.
• PR #1302:

P3 - Native trainer-to-vLLM weight synchronization

• Enable XPU support for trainer-to-vLLM weight synchronization by Q4'26.

Notes

• The device-agnostic core and attention changes are independent of the native weight-synchronization work and can be reviewed separately.
• No user-facing change on CUDA is intended at any stage.