flow_grpo

Flow-GRPO:
Training Flow Matching Models via Online RL

Changelog

2026-05-07

• Flow-GRPO is now supported in verl-omni! This provides a verl-style training framework for Flow-GRPO users.

** 2025-11-04**

• Adding GRPO-Guard.

2025-11-04

• Adding support for Bagel-7B.

2025-10-14

• Refactor FlowGRPO-Fast for compatibility with FlowGRPO, add CPS sampling and No-CFG training on SD3.

2025-08-15

• Adding support for Qwen-Image and Qwen-Image-Edit.

2025-08-15

• Thanks Jing Wang for adding Wan2.1. Training command

accelerate launch --config_file scripts/accelerate_configs/multi_gpu.yaml --num_processes=1 --main_process_port 29503 scripts/train_wan2_1.py --config config/grpo.py:general_ocr_wan2_1

2025-08-14

• Adding reward curve of Flow-GRPO-Fast vs. Flow-GRPO. In Pickscore reward, Flow-GRPO-Fast is comparable to Flow-GRPO with only 2 steps training.

2025-08-04

• Adding support for FLUX.1-Kontext-dev. For the counting task, we use Geneval reward to detect object counts and CLIP feature similarity to ensure consistency between the original and edited images. This implementation offers a runnable pipeline, but the training set contains only 800 samples. Making Flow-GRPO truly effective for editing tasks still requires further exploration by the community.

2025-07-31

• Adding Flow-GRPO-Fast.

2025-07-28

• Adding support for FLUX.1-dev.
• Adding support for CLIPScore as reward model.
• Introducing config.sample.same_latent to control whether the same noise is reused for identical prompts, addressing Issue #7.

2025-05-15

• We showcase image examples from three tasks and their training evolution at https://gongyeliu.github.io/Flow-GRPO. Check them out!
• We now provide an online demo for all three tasks at https://huggingface.co/spaces/jieliu/SD3.5-M-Flow-GRPO. You're welcome to try it out!

Model

Training Speed

To improve training efficiency, we provide a better set of parameters for Flow-GRPO. We found the following adjustments significantly accelerate training:

• No CFG during training or testing — the RL process effectively performs CFG distillation.
• Use the window mechanism from Flow-GRPO-Fast or MixGRPO — only train on partial steps.
• Adopt Coefficients-Preserving Sampling (CPS) — CPS provides a notable improvement on GenEval, and produces higher-quality samples. A typical setting is noise_level = 0.8, which works well without tuning for different models or step counts.

The figure below shows the test-set performance curves using GenEval and PickScore as rewards, where both training and evaluation are performed without CFG. The experiments are configured with geneval_sd3_fast_nocfg and pickscore_sd3_fast_nocfg, using scripts from scripts/multi_node/sd3_fast.

️ Over-optimization (GRPO-Guard)

To mitigates implicit over-optimization in flow matching, our team propose GRPO-Guard ( Project Page).

We first observe that the importance ratio exhibits an inherent bias:

1. Its mean is consistently below 1 and becomes significantly pronounced at low-noise steps (e.g., step 8 in SD3.5-M).

2. The variance varies notably across different steps.

Ideally, the importance ratio distribution should have a mean of 1 and stable variance. The clipping operation truncates overly confident positive or negative samples outside the region [1−ϵ,1+ϵ], ensuring stable gradient updates. However, the observed bias in the importance ratio disrupts this mechanism—gradients of positive samples are no longer properly constrained, leading the policy model into over-optimization. As a result, the proxy score continues to rise while the gold score declines, causing a severe degradation in image quality.

The biased ratio distributions are summarized in the table below.

To address this issue, GRPO-Guard introduces two mechanisms that effectively alleviate over-optimization:

• RatioNorm: Corrects the distributional bias of importance ratios and unifies their statistics across denoising steps.

• Gradient Reweight: Further reweights the gradients of different denoising steps based on RatioNorm, balancing their contributions and preventing excessive optimization under specific noise levels.

The following figure compares over-optimization between GRPO-Guard and FlowGRPO on text rendering tasks. GRPO-Guard maintains the same rising trend in proxy scores as FlowGRPO while preventing rapid declines in gold scores, thus preserving high image quality and diversity.

Start Training

After downloading the base model and setting up the reward model, run the following script to start training the GRPO-Guard for the SD3.5-M text rendering task.

# Master node
bash scripts/multi_node/sd3_grpo_guard.sh 0
# Other nodes
bash scripts/multi_node/sd3_grpo_guard.sh 1

Flow-GRPO-Fast

We propose Flow-GRPO-Fast, an accelerated variant of Flow-GRPO that requires training on only one or two denoising step per trajectory. For each prompt, we first generate a deterministic trajectory using ODE sampling. At a randomly chosen intermediate step, we inject noise and switch to SDE sampling to generate a group. The rest of the process continues with ODE sampling. This confines stochasticity to one or two steps, allowing training to focus solely on that steps. This few-step training idea was primarily proposed by Ziyang Yuan during our discussions in early June.

Flow-GRPO-Fast achieves significant efficiency gains:

• Each trajectory is trained only once or twice, significantly reducing the training cost.

• Sampling before branching requires only a single prompt without group expansion, further speeding up data collection.

Experiments on PickScore show that Flow-GRPO-Fast matches the reward performance of Flow-GRPO while offering faster training speed. The x-axis in the figure represents training epochs. Flow-GRPO-Fast with 2 training steps per iteration performs better than Flow-GRPO, while Flow-GRPO-Fast with only 1 training step per iteration performs slightly worse than Flow-GRPO. In both cases, compared to Flow-GRPO’s 10 training steps per iteration, the training process is significantly faster.

Please use scripts in scripts/multi_node/sd3_fast to run these experiments.

Quick Started

1. Environment Set Up

Clone this repository and install packages.

git clone https://github.com/yifan123/flow_grpo.git
cd flow_grpo
conda create -n flow_grpo python=3.10.16
pip install -e .

2. Model Download

To avoid redundant downloads and potential storage waste during multi-GPU training, please pre-download the required models in advance.

Models

• SD3.5: stabilityai/stable-diffusion-3.5-medium
• Flux: black-forest-labs/FLUX.1-dev

Reward Models

• PickScore:
  • laion/CLIP-ViT-H-14-laion2B-s32B-b79K
  • yuvalkirstain/PickScore_v1
• CLIPScore: openai/clip-vit-large-patch14
• Aesthetic Score: openai/clip-vit-large-patch14

3. Reward Preparation

The steps above only install the current repository. Since each reward model may rely on different versions, combining them in one Conda environment can cause version conflicts. To avoid this, we adopt a remote server setup inspired by ddpo-pytorch. You only need to install the specific reward model you plan to use.

GenEval

Please create a new Conda virtual environment and install the corresponding dependencies according to the instructions in reward-server.

OCR

Please install paddle-ocr:

pip install paddlepaddle-gpu==2.6.2
pip install paddleocr==2.9.1
pip install python-Levenshtein

Then, pre-download the model using the Python command line:

from paddleocr import PaddleOCR
ocr = PaddleOCR(use_angle_cls=False, lang="en", use_gpu=False, show_log=False)

Pickscore

PickScore requires no additional installation. Note that the original pickscore dataset corresponds to dataset/pickscore in this repository, containing some NSFW prompts. We strongly recommend using pickapic_v1_no_images_training_sfw, the SFW version of the Pick-a-Pic dataset, which corresponds to dataset/pickscore_sfw in this repository.

DeQA

Please create a new Conda virtual environment and install the corresponding dependencies according to the instructions in reward-server.

UnifiedReward

Since sglang may conflict with other environments, we recommend creating a new conda environment.

conda create -n sglang python=3.10.16
conda activate sglang
pip install "sglang[all]"

We use sglang to deploy the reward service. After installing sglang, please run the following command to launch UnifiedReward:

python -m sglang.launch_server --model-path CodeGoat24/UnifiedReward-7b-v1.5 --api-key flowgrpo --port 17140 --chat-template chatml-llava --enable-p2p-check --mem-fraction-static 0.85

ImageReward

Please install imagereward:

pip install image-reward
pip install git+https://github.com/openai/CLIP.git

4. Start Training

GRPO

Single-node training

# sd3
bash scripts/single_node/grpo.sh
# flux
bash scripts/single_node/grpo_flux.sh

# Master node
bash scripts/multi_node/sd3.sh 0
# Other nodes
bash scripts/multi_node/sd3.sh 1
bash scripts/multi_node/sd3.sh 2
bash scripts/multi_node/sd3.sh 3

# Master node
bash scripts/multi_node/flux.sh 0
# Other node
bash scripts/multi_node/flux.sh 1
bash scripts/multi_node/flux.sh 2
bash scripts/multi_node/flux.sh 3

For Flow-GRPO-Fast, please use scripts/multi_node/flux_fast.sh. See the W&B logs for Geneval (with geneval_flux_fast in the config) and PickScore (with pickscore_flux_fast in the config).

Please first download generated_images.zip and extract it into the counting_edit directory. You can also use the scripts in the counting_edit directory to generate the data yourself.

Please install diffusers from the main