Solve Visual Understanding with Reinforced VLMs
Since the introduction of Deepseek-R1, numerous works have emerged focusing on reproducing and improving upon it. In this project, we propose VLM-R1, a stable and generalizable R1-style Large Vision-Language Model.
Specifically, for the task of Referring Expression Comprehension (REC), we trained Qwen2.5-VL using both R1 and SFT approaches. The results reveal that, on the in-domain test data, the performance of the SFT model shows little change compared to that of the R1 model base model when the number of training steps is relatively small (100–600 steps), while the R1 model shows a steady improvement (as shown at the left of the figure below). More importantly, on the out-of-domain test data, the SFT model's performance deteriorates slightly as the number of steps increases. Nevertheless, the RL model generalizes its reasoning ability to the out-of-domain data (as shown at the right of the figure below).
* We found previous REC SFT exps used a mismatch pixel config. Therefore, we re-run the study with the correct config on a more complex out-of-domain data. See our findings for details.
This repository supports:
Full Fine-tuning for GRPO: see run_grpo_rec.shFreeze Vision Modules: set freeze_vision_modules as true in the script.LoRA Fine-tuning for GRPO: see run_grpo_rec_lora.shMulti-node Training: see multinode_training_demo.shMulti-image Input Training: see run_grpo_gui.shFor your own data: see hereVarious VLMs: see How to add a new model, now we support QwenVL and InternVL2025-08-29: We have further optimized the VLM-R1 series models based on JD's latest open-source inference framework xllm (github is here). The TTFT (Time to First Token) has been reduced by 50% compared to vllm-ascend, and the overall throughput has increased by 127% compared to vllm-ascend. Please refer to ascend_inference/910B/xllm/README.md for more details.
2025-08-22: We have adapted the VLM-R1 series models to Huawei Ascend Atlas 800T A2 and Atlas 300I Duo series using the vllm-ascend framework, further expanding the deployment scenarios and hardware compatibility of the model series. Please refer to ascend_inference/910B/vllm_ascend/README.md and ascend_inference/300IDuo/README.md for more details.
2025-06-26: We introduce a post-resize operation for the bounding box for QwenVL (both training and evaluation) and the results are improved slightly.
2025-04-16: We have updated the codebase to improve functionality and maintain unified implementation. Specifically, the REC process is now integrated into grpo_jsonl.py for consistency across tasks. Additionally, we introduce a new parameter, is_reward_customized_from_vlm_module, which enables the use of customized reward functions defined within the VLM module. When set to true, the reward logic is handled in either QwenVL2Module or InternVLModule, depending on the selected model. Furthermore, the training log has been enhanced to provide more detailed output for easier monitoring and debugging.
2025-04-11: We release the technical report of VLM-R1, summarizing our main results and insights.
2025-04-03: We add the odLength, weighted_sum, and cosine reward used in OVD task, please refer our blog post and findings to the details of the reward usage and see grpo_jsonl.py for code implementation.
2025-03-24: We release the findings of VLM-R1-OVD.
2025-03-23: We release the VLM-R1-OVD model weights and demo, which shows the state-of-the-art performance on OVDEval. Welcome to use it.
2025-03-20: We achieved SOTA results on OVDEval with our RL-based model, outperforming SFT baselines and specialized object detection models. Read our blog post for details on how reinforcement learning enhances object detection performance.
2025-03-17: Our VLM-R1 Math model reaches the top of the Open-Compass Math Leaderboard (under 4B parameters). We have released the checkpoint.
2025-03-15: We support multi-image input data. Check the format of multi-image input here. We also provide an example of multi-image script run_grpo_gui.sh, see here for details.
2025-03-13: We support InternVL for GRPO. See run_grpo_rec_internvl.sh for details. The annotation json files used in InternVL are here. If you want to add your new model, please refer to How to add a new model.
2025-03-02: We support LoRA Fine-tuning for GRPO. See run_grpo_rec_lora.sh for details.
2025-02-27: We support the number of iterations per batch and epsilon value for clipping in the original GRPO algorithm with args: --num_iterations and --epsilon.
2025-02-25: We support multi-node training for GRPO. See multinode_training_demo.sh for details.
2025-02-21: We release the checkpoint of the VLM-R1 REC model.
2025-02-20: We release the script for general data loading.
2025-02-19: We incorporate an explanation of the SFT method.
2025-02-17: We release the VLM-R1 REC Demo on Hugging Face Spaces.
2025-02-15: We release the VLM-R1 repository and GRPO training script.
OVD: Trained with VLM-R1, our Open-Vocabulary Detection (OVD) model achieves the state-of-the-art performance on OVDEval.Math: Through VLM-R1 training, our math model focuses on multimodal reasoning tasks and has achieved Top1 on the OpenCompass Multi-modal Reasoning Leaderboard among models < 4B.REC: Trained with VLM-R1, our Referring Expression Comprehension (REC) model showcases the superior performance on out-of-domain data and a series of reasoning-grounding tasks.GUI: Trained with VLM-R1, our GUI Defect Detection model outperforms both base and SFT models by achieving the best accuracy and improved generalization across both defective and clean screens.conda create -n vlm-r1 python=3.10
conda activate vlm-r1
bash setup.sh
<your_image_root>.data_paths and image_folders in the run_scripts/run_grpo_rec.sh file.# These jsonl files are included in the annotation files at step 2.
# Note: please use jsonl files instead of json files.
data_paths="path/to/refcoco_train.jsonl:path/to/refcocop_train.jsonl:path/to/refcocog_train.jsonl"
image_folders="path/to/coco:path/to/coco:path/to/coco"
bash run_scripts/run_grpo_rec.sh[!NOTE] If you encounter 'CUDA out of memory' error, you can try to reduce the
per_device_train_batch_size.
For multi-node training, please refers to multinode_training_demo.sh.
We use LLaMA-Factory to train the SFT model.
git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]"
LLaMA-Factory/data directory and the yaml file in the LLaMA-Factory/examples/train_full directory.llamafactory-cli train examples/train_full/qwen2_5_vl_full_sft.yaml
The jsonl has the format as follows:
{
"id": 1,
"image": "Clevr_CoGenT_TrainA_R1/data/images/CLEVR_trainA_000001_16885.png",
"conversations": [
{"from": "human", "value": "<image>What number of purple metallic balls are there?"},
{"from": "gpt", "value": "0"}
]
}
``
No open issues yet, or sync has not completed.