百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
L

lm-human-preference-details

> 编程语言
开源

OAI 2019 代码库的 RLHF 实现细节

198 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

OAI 2019 代码库的 RLHF 实现细节

lm-human-preference-details

This repo aims to make a blog post similar to The 37 Implementation Details of Proximal Policy Optimization but for RLHF techniques used in https://github.com/openai/lm-human-preferences.

Warning This repo is a WIP made public because it's easier for me to share pointers with collaborators. I'll remove this warning when the repo is ready for public consumption.

The goal of the repo is 1) to provide a simple-to-read and minimal reference implementation of RLHF and 2) to create rigorous benchmarks and to match the learning curves of openai/lm-human-preferences.

This repo is just for educational / learning purposes. For more advanced users, https://github.com/lvwerra/trl would be a great choice.

Get started

bash
poetry install
poetry shell
accelerate launch \
    --num_processes 8 \
    lm_human_preference_details/train_both_accelerate.py \
    --reward.track --policy.track
accelerate launch \
    --num_processes 8 \
    lm_human_preference_details/train_both_accelerate.py \
    --reward.track \
    --reward.label_dataset=descriptiveness/offline_5k.json \
    --policy.track

You can also run stuff individually. For example, to train the reward model, run

bash
accelerate launch \
    --num_processes 8 \
    lm_human_preference_details/train_reward_accelerate.py \
    --track

to train the policy model, run

bash
accelerate launch \
    --num_processes 8 \
    lm_human_preference_details/train_policy_accelerate.py \
    --track

⚠️ NOTE: You can install the latest torch or jax with the following command:

poetry run pip install torch==2.0.1
poetry run pip install "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
poetry run pip install git+https://github.com/huggingface/huggingface_hub@main

Current status

Currently, for reproduction, I used the same dataset, data processing pipeline, hyperparameters, and initial model architecture and weights (gpt2 124M pretrained model).

The following charts show the learning curves of various metrics for sentiment and descriptiveness tasks, each with 10 random seeds of commit 46725b.

…

Wandb report is available at https://wandb.ai/costa-huang/cleanrl/reports/Regression-Report-train_policy_accelerate--Vmlldzo1MTEwMzQw. Feel free to check out the logs of the runs for sample outputs.

Sentiment

  • https://wandb.ai/openrlbenchmark/lm-human-preferences/runs/ckg9d00z/logs
…
  • https://wandb.ai/openrlbenchmark/lm_human_preference_details/runs/38v19l2k
…

Descriptiveness

  • https://wandb.ai/openrlbenchmark/lm-human-preferences/runs/vx0cjya8/logs
…
  • https://wandb.ai/openrlbenchmark/lm_human_preference_details/runs/e3rysbw8
    30052 query: she shivered.
    30053 response: She looked up at the old man who had lain with her, and he gave her a
    30054 kiss.
    30058 score: 2.936002731323242, kl: 3.442370891571045, total reward: 1.6455250978469849 
    
    30023 query: asked jason with frustration apparent in his strained voice.
    30024 response: He was dressed in a suit and tie with a white button down shirt
    30025 and a white knitted hat.
    30028 score: 4.891026973724365, kl: 10.192875862121582, total reward: 1.0853893756866455 

Learning curves of openai/lm-human-preferences

Wandb report is here: https://wandb.ai/costa-huang/cleanrl/reports/Regression-Report-124M--Vmlldzo0ODM3NTI5

…

Acknowledgement

This work is supported by Hugging Face's Big Science A100 cluster.

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言