[Question] Why does VecNormalize normalize rewards by returns instead of rewards?

❓ Question

I just read the code of VecNormalize and was surprised by the way the reward normalization is computed. I had assumed the wrapper to keep track of past rewards, storing their running mean and variance. New rewards would then be normalized using this statistic. Instead the wrapper appears to normalize the rewards using statistics of the returns, i.e. the discounted sums over rewards. This is done in the _update_reward function. For environments with only positive rewards (like CartPole) the value of the return increases as the episode progresses, particularly for large values of gamma, while the value of the reward does not depend on the time step within the episode. Normalizing rewards by statistics over returns seems counterintuitive to me, so I wanted to ask if this is a bug or intended behaviour

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.