#1243·OpenRLHF

overlong_penalty should exclude tool response tokens from length calculation (use action_mask/action_ranges)

Author: thevasudevguptaCreated May 26, 2026Updated Jun 18, 2026

when doing "agentic" training (i.e. training with tools), overlong buffer uses the experience.response length as per this following line:

https://github.com/OpenRLHF/OpenRLHF/blob/6c6056daa522e2216466f6e0351cbe453434e185/openrlhf/trainer/ppo_utils/length_penalty.py#L45

response length includes tokens from tool response as per following line:

https://github.com/OpenRLHF/OpenRLHF/blob/3a9815c51aa7e985983eacd06504aaf6db0bbe6b/openrlhf/trainer/ppo_utils/samples_generator.py#L277

given tool responses can't be optimised by model, this can add a bit of noise to "agentic" training and worse can discourage model to use tools.

we should compute response length using action_mask or action_ranges inside overlong penalty function.

Happy to create PR to fix this. Please let me know if you would like me to.