`log_prob` and `old_log_prob` differ in on-policy setting when rollout and actor micro batch per gpu differ

When following the official recommendation to set rollout.log_prob_micro_batch_size_per_gpu = 2 * actor.ppo_micro_batch_size_per_gpu, log_prob and old_log_prob are different even in an on-policy setting:

actor.ppo_epochs = 1 and train_batch_size = actor.ppo_mini_batch_size.

In this setting, the training should be on-policy, but pg_clipfrac in the logs is non-zero, indicating that the logits are different.

A simple workaround is to set:

rollout.log_prob_micro_batch_size_per_gpu = actor.ppo_micro_batch_size_per_gpu

Another possible workaround for the on-policy case is to directly set:

old_log_prob = log_prob

This was done in #3119, but seems to have been removed in newer versions. Also, this workaround is not valid for off-policy training.

Is this caused by kernels that are not batch-invariant? Is there a way to fix this without significantly affecting performance?