OpenRLHF

---

[ English | 中文 | 日本語 ]

OpenRLHF is the first high-performance, production-ready open-source RLHF framework that combines Ray + vLLM distributed architecture with a unified agent-based design paradigm for scalable and extensible reinforcement learning from human feedback.

Learn More: Documentation | Slides | Technical Report | Video

New Backend: Molt brings an Automodel-powered backend to OpenRLHF that's more powerful than DeepSpeed — scaling RL training to hundreds of billions of parameters while keeping the same familiar, elegant OpenRLHF workflow.

Table of Contents

• ️ News
• ️ Architecture Foundation - Ray + vLLM + DeepSpeed distributed infrastructure
• Design Paradigm - Unified agent-based execution pipeline
• RL Algorithms - PPO, REINFORCE++, GRPO, RLOO
• Features Overview - Complete RLHF pipeline capabilities
• Quick Start - Installation and typical workflow
• Training Guide - SFT, Reward Model, RL Training
• Single-Turn Agent - Custom reward functions
• Multi-Turn Agent - Complex environments
• Advanced Topics - LoRA, performance tuning

---

News

Show News

• [2026/9] OpenRLHF supports FlashREINFORCE — critic-free, single-rollout asynchronous RL for agentic language models, as a composition of configs (--algo.advantage.estimator flash_reinforce, a binary-KL trust region on the vLLM logprobs, sample-mean aggregation). Training script: train_flash_reinforce_ray_agent_async.sh
• [2026/4] OpenRLHF 0.10 adds Multi-Turn VLM RL — multi-step interactions with images in both prompts and environment feedback (e.g. screenshots). Example: vlm_multiturn_agent.py
• [2026/4] OpenRLHF 0.10 adds VLM (Vision-Language Model) RLHF support — train VLMs like Qwen3.5 with image inputs end-to-end. Training script: train_vlm_math_hybrid_engine.sh
• [2026/2] ProRL V2 uses REINFORCE++-baseline to train a state-of-the-art 1.5B reasoning model with prolonged RL training. Training script: train_prorlv2_math_hybrid_engine.sh
• [2025/10] ScaleRL validates the effectiveness of REINFORCE++-baseline in large-scale training scenarios. Releases REINFORCE++ slides
• [2025/6] Magistral uses the method quite similar to REINFORCE++-baseline to train the reasoning models.
• [2025/5] MARTI has been released as a fork of OpenRLHF. It is designed to train LLM-based multi-agent systems using RL, by integrating centralized multi-agent interactions with distributed policy training.
• [2025/5] OpenRLHF 0.8.0 supports async RLHF training via --train.async_enable and async agent RLHF via --train.agent_func_path. See train_reinforce_baseline_ray_agent_async.sh for a runnable example.
• [2025/4] Post the blog Accelerating RLHF with vLLM, Best Practice from OpenRLHF
• [2025/4] Clean OpenRLHF: Refactored the source code based on Single Controller and Unified Packing Samples
• [2025/3] The CMU Advanced Natural Language Processing Spring 2025 course uses OpenRLHF as the RLHF framework teaching case.
• [2025/2] Logic-RL and PRIME demonstrate that REINFORCE++ is more stable in training compared to GRPO and faster than PPO.
• [2025/2] LMM-R1 is a fork of OpenRLHF, aimed at providing high-performance RL infrastructure for reproduction of DeepSeek-R1 on multimodal tasks.
• [2025/2] MIT & Microsoft proposed the On the Emergence of Thinking in LLMs I: Searching for the Right Intuition using OpenRLHF
• [2025/1] HKUST reproduced the DeepSeek-R1-Zero and DeepSeek-R1 training on small models using OpenRLHF
• [2024/12] We "proposed" the REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models.
• [2024/12] We analyzed the PPO, REINFORCE++, GRPO and RLOO in the Notion Blogpost.
• [2023/8] OpenRLHF was open-sourced.

---

️ Architecture Foundation: Ray + vLLM Distribution

OpenRLHF is the first RLHF framework built on Ray + vLLM distributed architecture, orchestrating multiple components across GPUs efficiently:

Core Infrastructure Components

Ray - Distributed Scheduler and Controller
OpenRLHF leverages Ray for efficient distributed scheduling. It separates the Actor, Reward, Reference, and Critic models across different GPUs, enabling scalable training for models up to 70B+ parameters.

Hybrid Engine Scheduling: All models and vLLM engines can share GPU resources—minimizing idle time and maximizing GPU utilization. This allows running full RLHF pipelines on limited hardware.

vLLM - High-Performance Inference Engine
RLHF training spends 80% of the time on sample generation. Powered by vLLM with Auto Tensor Parallelism (AutoTP) and Pipeline Parallelism (PP), OpenRLHF delivers high-throughput, memory-efficient generation.

DeepSpeed - Memory-Efficient Training
Built on DeepSpeed ZeRO-3, deepcompile, AutoTP, and RingAttention. Enables large model training without heavyweight frameworks while working directly with HuggingFace models.

Transformers - Model Interface
Native integration with HuggingFace Transformers for seamless model loading, state management, and fine-tuning of pretrained models.

NCCL / CUDA IPC - High-Speed Communication
Efficient inter-GPU communication for distributed training and inference.

---

Design Paradigm: Agent-Based Execution

On top of the Ray distributed architecture, OpenRLHF is the first RLHF framework to implement a unified agent-based paradigm. Every training run—whether standard PPO or complex multi-turn reasoning—follows a consistent agent execution pipeline.

Why Agent-Based?

OpenRLHF unifies generation and training through token-in-token-out agent execution, ensuring perfect consistency, easy single/multi-turn extension, and zero text-level mismatches.

Agent Architecture

…

Core Design Principles

Show core design principles Principle Description Benefit Token-in-Token-out All sampling produces token-level trajectories Zero text-level mismatch Unified Interface Same AgentExecutorBase API for all modes Switch modes with one flag Algorithm-Agnostic RL algorithms (PPO, REINFORCE++, etc.) are decoupled from agent executors Any algorithm works with any mode Extensible Plug in custom rewards/environments easily Rapid experimentation Production-Ready Sync/Async/Hybrid Engine support From research to deployment

Two Execution Modes (Orthogonal to RL Algorithms)

The agent execution mode is independent of the RL algorithm you choose. You can use any algorithm (PPO, REINFORCE++, GRPO, etc.) with any execution mode:

Mode Use Cases Interface Complexity Single-Turn Standard RLHF, custom reward functions Optional reward_func() ⭐ Default (99% use cases) Multi-Turn Multi-step reasoning, interactive environments reset() + step() ⭐⭐ Advanced

---

State-of-the-Art RL Algorithms

OpenRLHF implements PPO, REINFORCE++, REINFORCE++-baseline, GRPO, RLOO with advanced optimization tricks inspired by practical guides and community best practices.

Key Design: RL algorithms are decoupled from agent execution modes. All algorithms work seamlessly with both single-turn and multi-turn agent executors, running through the unified token-in-token-out pipeline for consistent behavior.

Show algorithm comparison table Algorithm --algo.advantage.estimator Key Feature Best Use Case PPO (default) Full critic network Stable training, proven results REINFORCE++ reinforce PPO tricks without critic Efficient training, less memory REINFORCE++-baseline reinforce_baseline Mean reward baseline Reasoning tasks (RLVR), robust to reward scales RLOO rloo Per-token KL + PPO-clip Multi-sample training GRPO group_norm Group normalization Batch-based training Dr. GRPO dr_grpo Simplified GRPO Removes local /std norm FlashREINFORCE flash_reinforce Critic-free single-rollout RL: batch-mean baseline, binary-KL trust region on the vLLM logprobs Async agentic RL with one rollout per prompt (script)

References: Zhihu article | Notion best practices

---

Comprehensive Features

OpenRLHF provides a complete RLHF pipeline with agent-based flexibility:

Agent-Based RL Training (Core Innovation)

Show agent-based RL training details

Single-Turn Mode (Default - 99% of use cases)

• One-shot generation per prompt
• Works with all RL algorithms: PPO, REINFORCE++/baseline/GRPO/RLOO
• Custom reward functions (--reward.remote_url)
• Hybrid Engine for maximum GPU