Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
L

lm-human-preference-details

> 编程语言
Open source

RLHF implementation details of OAI's 2019 codebase

198 stars0 likes0 views
WebsiteGitHub

About

RLHF implementation details of OAI's 2019 codebase

lm-human-preference-details

This repo aims to make a blog post similar to The 37 Implementation Details of Proximal Policy Optimization but for RLHF techniques used in https://github.com/openai/lm-human-preferences.

Warning This repo is a WIP made public because it's easier for me to share pointers with collaborators. I'll remove this warning when the repo is ready for public consumption.

The goal of the repo is 1) to provide a simple-to-read and minimal reference implementation of RLHF and 2) to create rigorous benchmarks and to match the learning curves of openai/lm-human-preferences.

This repo is just for educational / learning purposes. For more advanced users, https://github.com/lvwerra/trl would be a great choice.

Get started

bash
poetry install
poetry shell
accelerate launch \
    --num_processes 8 \
    lm_human_preference_details/train_both_accelerate.py \
    --reward.track --policy.track
accelerate launch \
    --num_processes 8 \
    lm_human_preference_details/train_both_accelerate.py \
    --reward.track \
    --reward.label_dataset=descriptiveness/offline_5k.json \
    --policy.track

You can also run stuff individually. For example, to train the reward model, run

bash
accelerate launch \
    --num_processes 8 \
    lm_human_preference_details/train_reward_accelerate.py \
    --track

to train the policy model, run

bash
accelerate launch \
    --num_processes 8 \
    lm_human_preference_details/train_policy_accelerate.py \
    --track

⚠️ NOTE: You can install the latest torch or jax with the following command:

poetry run pip install torch==2.0.1
poetry run pip install "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
poetry run pip install git+https://github.com/huggingface/huggingface_hub@main

Current status

Currently, for reproduction, I used the same dataset, data processing pipeline, hyperparameters, and initial model architecture and weights (gpt2 124M pretrained model).

The following charts show the learning curves of various metrics for sentiment and descriptiveness tasks, each with 10 random seeds of commit 46725b.

…

Wandb report is available at https://wandb.ai/costa-huang/cleanrl/reports/Regression-Report-train_policy_accelerate--Vmlldzo1MTEwMzQw. Feel free to check out the logs of the runs for sample outputs.

Sentiment

  • https://wandb.ai/openrlbenchmark/lm-human-preferences/runs/ckg9d00z/logs
…
  • https://wandb.ai/openrlbenchmark/lm_human_preference_details/runs/38v19l2k
…

Descriptiveness

  • https://wandb.ai/openrlbenchmark/lm-human-preferences/runs/vx0cjya8/logs
…
  • https://wandb.ai/openrlbenchmark/lm_human_preference_details/runs/e3rysbw8
    30052 query: she shivered.
    30053 response: She looked up at the old man who had lain with her, and he gave her a
    30054 kiss.
    30058 score: 2.936002731323242, kl: 3.442370891571045, total reward: 1.6455250978469849 
    
    30023 query: asked jason with frustration apparent in his strained voice.
    30024 response: He was dressed in a suit and tie with a white button down shirt
    30025 and a white knitted hat.
    30028 score: 4.891026973724365, kl: 10.192875862121582, total reward: 1.0853893756866455 

Learning curves of openai/lm-human-preferences

Wandb report is here: https://wandb.ai/costa-huang/cleanrl/reports/Regression-Report-124M--Vmlldzo0ODM3NTI5

…

Acknowledgement

This work is supported by Hugging Face's Big Science A100 cluster.

Issues· 0 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Python

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 18, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言