Early Stopping When Approximate KL Divergence is Negative in PPO
Author: batuCreated Apr 2, 2019Updated Sep 16, 2025
First off thank you very much for this project! It has been immensely helpful.
While going through the PPO code I saw the following lines: https://github.com/openai/spinningup/blob/97c8c342c45e5bb51005a8515df23ba9c48f0782/spinup/algos/ppo/ppo.py#L232-L238
Here we only break the iteration if the divergence is higher than the threshold and is positive. Shouldn't we also break if the divergence is higher than the threshold and is negative given that we are using the approximate KL divergence?
Let me know if this is a bug and I can submit a pull request!
Source: openai/spinningup