RLHF implementation details of OAI's 2019 codebase
RLHF implementation details of OAI's 2019 codebase
This repo aims to make a blog post similar to The 37 Implementation Details of Proximal Policy Optimization but for RLHF techniques used in https://github.com/openai/lm-human-preferences.
Warning This repo is a WIP made public because it's easier for me to share pointers with collaborators. I'll remove this warning when the repo is ready for public consumption.
The goal of the repo is 1) to provide a simple-to-read and minimal reference implementation of RLHF and 2) to create rigorous benchmarks and to match the learning curves of openai/lm-human-preferences.
This repo is just for educational / learning purposes. For more advanced users, https://github.com/lvwerra/trl would be a great choice.
poetry install
poetry shell
accelerate launch \
--num_processes 8 \
lm_human_preference_details/train_both_accelerate.py \
--reward.track --policy.track
accelerate launch \
--num_processes 8 \
lm_human_preference_details/train_both_accelerate.py \
--reward.track \
--reward.label_dataset=descriptiveness/offline_5k.json \
--policy.trackYou can also run stuff individually. For example, to train the reward model, run
accelerate launch \
--num_processes 8 \
lm_human_preference_details/train_reward_accelerate.py \
--trackto train the policy model, run
accelerate launch \
--num_processes 8 \
lm_human_preference_details/train_policy_accelerate.py \
--track⚠️ NOTE: You can install the latest torch or jax with the following command:
poetry run pip install torch==2.0.1
poetry run pip install "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
poetry run pip install git+https://github.com/huggingface/huggingface_hub@mainCurrently, for reproduction, I used the same dataset, data processing pipeline, hyperparameters, and initial model architecture and weights (gpt2 124M pretrained model).
The following charts show the learning curves of various metrics for sentiment and descriptiveness tasks, each with 10 random seeds of commit 46725b.
…Wandb report is available at https://wandb.ai/costa-huang/cleanrl/reports/Regression-Report-train_policy_accelerate--Vmlldzo1MTEwMzQw. Feel free to check out the logs of the runs for sample outputs.
………30052 query: she shivered.
30053 response: She looked up at the old man who had lain with her, and he gave her a
30054 kiss.
30058 score: 2.936002731323242, kl: 3.442370891571045, total reward: 1.6455250978469849
30023 query: asked jason with frustration apparent in his strained voice.
30024 response: He was dressed in a suit and tie with a white button down shirt
30025 and a white knitted hat.
30028 score: 4.891026973724365, kl: 10.192875862121582, total reward: 1.0853893756866455 openai/lm-human-preferencesWandb report is here: https://wandb.ai/costa-huang/cleanrl/reports/Regression-Report-124M--Vmlldzo0ODM3NTI5
…This work is supported by Hugging Face's Big Science A100 cluster.
No open issues yet, or sync has not completed.