Feature request: reward-hacking onset monitoring hooks during PPO/GRPO training

Feature request

Add optional monitoring hooks to OpenRLHF's PPO/GRPO trainers that track signals correlated with reward-hacking onset — KL-vs-reference acceleration, entropy-collapse trend, and advantage-distribution drift (Wasserstein vs. a rolling baseline) — logged alongside existing training metrics. Related code: https://github.com/Aarav500/flight-recorder (Apache-2.0), which implements these extractors for GRPO already.

Motivation

Reward hacking / reward-model overoptimization is exactly where RLHF training goes wrong in practice. I built a benchmark specifically testing whether these signals give early warning (paper: "Audit or Early Warning? A Benchmark for Online Detectors of Reward Hacking") — honest finding: they reliably audit hacking after the fact but don't reliably give early warning at a usable false-positive rate. Seemed directly relevant for anyone relying on these signals during OpenRLHF training.

Your contribution

Proposing before attempting a PR. Happy to share the Flight Recorder code/methodology and adapt it into an OpenRLHF-style hook if there's interest.