MOSS-RLHF

MOSS-RLHF

Congratulations We received the best paper award at NIPS 2023 Workshop on Instruction Tuning and Instruction Following!

庆祝, "Secrets of RLHF in Large Language Models Part I: PPO" 荣获 NIPS 2023 Workshop on Instruction Tuning and Instruction Following 最佳论文奖！！！

### *MOSS-RLHF [Home page]*

"Secrets of RLHF in Large Language Models Part I: PPO" [Technical report I]

"Secrets of RLHF in Large Language Models Part II: Reward Modeling" [Technical report II]

Breaking News

Mon, 15. January 2024. We have released the code for training the reward model and the annotated hh-rlhf dataset(hh-rlhf-strength-cleaned)!

Fri, 12. January 2024. We have released the second paper "Secrets of RLHF in Large Language Models Part II: Reward Modeling"!

News

Wed, 12. July 2023. We have released Chinese reward model based OpenChineseLlama-7B! moss-rlhf-reward-model-7B-zh <br>

Thu, 13. July 2023. We have released English reward model and SFT model based Llama-7B! moss-rlhf-reward-model-7B-en

moss-rlhf-sft-model-7B-en <br>

Wait a minute ! Thu, 14. July 2023. We have released English policy model after aligning with RLHF! moss-rlhf-policy-model-7B-en <br>

Open-source List

RL related

• Open source code for RL training in large language models.
• A 7B Chinese reward model based on openChineseLlama.
• A 7B English reward model based on Llama-7B.
• SFT model for English.
• Policy model for English after RLHF.

RM related

• Open source code for reward model training in large language models.

• HH-RLHF dataset with preference strength annotation.

• HH-RLHF validation set cleaned by GPT-4.

• ...

Introduction

Due to the challenges of reward design, environment interaction, and agent training, coupled with huge trial and error cost of large language models, there is a significant barrier for AI researchers to motivate the development of technical alignment and safe landing of LLMs. The stable training of RLHF has still been a puzzle. In this technical report, we intend to help researchers to train their models stably with human feedback.

Contributions are summarized as follows:

1. We release competitive Chinese and English reward models, respectively, which have good cross-model generalization ability, alleviating the cost of relabeling human preference data;
2. We conduct in-depth analysis on the inner workings of PPO algorithm and propose the PPO-max algorithm to ensure stable model training;
3. We release the complete PPO-max codes to ensure that the LLMs in the current SFT stage can be better aligned with humans.

Requirements & Setup

This repository works on Python 3.8 and PyTorch 1.13.1.

We recommend using the conda virtual environment to run the code.

Step 1: Create a new Python virtual environment

conda update conda -n base -c defaults
conda create -n rlhf python=3.8
conda activate rlhf

Step 2: Install PyTorch and TensorBoard

conda install pytorch==1.13.1 pytorch-cuda=11.7 tensorboard -c pytorch -c nvidia

Step 3: Install the remaining dependencies

conda install datasets accelerate safetensors chardet cchardet -c huggingface -c conda-forge
pip3 install transformers sentencepiece einops triton==1.0.0 rouge jionlp==1.4.14 nltk sacrebleu cpm_kernels

apt install libaio-dev
DS_BUILD_OPS=1 pip install deepspeed

✨ Start training your own model!

Training PPO model

Run code in a few steps.

Step 1: Recover Reward model weights

We can not directly release the full weight of the reward model because of protocol restrictions. You can merge the diff weight with original Llama-7B to recover the reward model we used.

We upload the diff models, thanks to tatsu-lab, you can recover the reward model follow these steps:

…

Step 2: Select your own SFT model.

Because of some limitations, we can not release the Chinese SFT model (Currently). You can use your own SFT model, or a strong base model instead of our SFT model.

Step 3: Start training

Run the command below.

# For Chinese:
# You need to use your own sft model currently.
bash train_ppo_zh.sh

# For English:
# We have loaded the sft model and reward model to huggingface.
bash train_ppo_en.sh

Training reward model

To train the reward model, you need to specify the initial model (--hf_model_name_or_path) for the reward model (e.g., meta-llama/Llama-2-7b-hf) and preference dataset(--data_path) (such as hh-rlhf, or you can use our provided annotated hh-rlhf which has a format consistent with the training code), and run the command below.

# annotated dataset: https://huggingface.co/datasets/fnlp/hh-rlhf-strength-cleaned
# Assuming you have specified the --hf_model_name_or_path and --data_path parameters.
bash train_rm.sh

Citation

…

…