#563·cleanrl

Optional monitoring callback for reward-hacking / training-instability signals

Author: Aarav500Created Jul 10, 2026Updated Jul 12, 2026

CleanRL's algorithms currently log standard training metrics (loss, entropy, KL) but nothing that flags reward hacking or instability onset during a run. I'd like to propose an optional, standalone cleanrl_utils/monitors.py — not imported by any existing algorithm file by default — that users can opt into by adding a few lines to their own script. It tracks KL-vs-frozen-reference acceleration, entropy-collapse trend, and advantage-distribution drift (Wasserstein vs. a rolling baseline), and flags onset events.

This comes out of work on two reward-hacking-detection benchmarks I've built (RHOB, Flight Recorder) — happy to share results/methodology if useful context. Wanted to check whether this is something you'd want in the repo (even just as an example under docs/ or benchmark/) before spending time on a PR, given CleanRL's single-file philosophy.