Droping experiences should also consider standard deviation when using non-binary rewards

The current dynamic filtering strategy is based on the reward average, which basically filters out samples that are too hard or easy.

See the code from trainer/ppo_utils/samples_generator.py below:

# Drop experiences if the average score falls outside the allowed range.
if dynamic_filtering and all(e.scores is not None for e in experiences):
    scores = [e.scores[0].item() for e in experiences]
    avg_reward = sum(scores) / len(scores)
    min_r, max_r = self.args.algo.dynamic_filtering_range
    if not (min_r < avg_reward < max_r):
        logger.info(
            f"Filtered out: avg_reward={avg_reward:.2f}, threshold=({min_r:.2f}, {max_r:.2f}), scores={[f'{s:.2f}' for s in scores]}"
        )
        experiences = []

However, in fact, we need to filter out samples with standard deviation of the rewards to achieve zero-gradient signal filtering for GRPO based on https://arxiv.org/abs/2503.14476. For binary rewards, the averaging strategy works. When using non-binary rewards, e.g., token-based F1, we need to further drop samples with zero standard deviation as they just waste more compute.