overlong_penalty should exclude tool response tokens from length calculation (use action_mask/action_ranges)
Author: thevasudevguptaCreated May 26, 2026Updated Jun 18, 2026
when doing "agentic" training (i.e. training with tools), overlong buffer uses the experience.response length as per this following line:
response length includes tokens from tool response as per following line:
given tool responses can't be optimised by model, this can add a bit of noise to "agentic" training and worse can discourage model to use tools.
we should compute response length using action_mask or action_ranges inside overlong penalty function.
Happy to create PR to fix this. Please let me know if you would like me to.
Source: OpenRLHF/OpenRLHF