GRPO: vLLM importance-sampling ratio is biased when top_p/top_k/min_p truncate sampling

When top_p < 1, top_k > 0 or min_p is set, GRPO's vLLM importance-sampling correction compares two log-probabilities computed over different distributions, so the ratio is biased. vLLM returns log-probs renormalized over the truncated (nucleus) distribution, while the trainer recomputes them over the full vocabulary.

Two other RL frameworks have now shipped a training-side fix for exactly this, and the native vLLM fix is merged but not yet in a release TRL supports.

Where the mismatch is

Generation side asks vLLM for post-processor log-probs, in both colocate and server mode:

https://github.com/huggingface/trl/blob/main/trl/generation/vllm_generation.py#L365

Training side divides by temperature and takes a full-vocab log-softmax:

https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_trainer.py#L1530-L1534

processed_logprobs means "after every logits processor", which includes top-p/top-k truncation followed by renormalization over the surviving tokens. Temperature is handled symmetrically (line 1532 mirrors it, and that symmetry was the fix for #4159); truncation is not. For a sampled token that survives the nucleus, vLLM's log-prob is therefore higher than the trainer's by -log(sum of surviving probability mass), which is a per-token, systematically negative bias in log π_θ - log π_behaviour.

importance_sampling_ratio multiplies the per-token loss directly:

https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_trainer.py#L3208-L3209

so the effect is a silent, token-dependent down-weighting of the policy gradient rather than a crash.

Scope

• Defaults are safe: top_p=1.0, top_k=0, min_p=None, so nothing is truncated and the two distributions agree.

• AsyncGRPOTrainer is affected more severely. Server launch passes --logprobs-mode processed_logprobs, and those generator log-probs are not an optional correction there: they are the denominator of the policy ratio itself, with no flag guarding them.

  python复制

  log_ratio = log_probs - old_log_probs   # async_grpo_trainer.py#L945
  coef_1 = torch.exp(log_ratio)

  AsyncGRPOConfig exposes top_p/top_k/min_p like GRPO, and it logs no sampling_logp_difference equivalent, so there is nothing to notice the drift with.

• Affected as soon as a user sets any of them, which top_p=0.9/top_k=50 recipes sometimes do.

• Only active with vllm_importance_sampling_correction=True.

• Same logprobs_mode="processed_logprobs" is set in trl/scripts/vllm_serve.py#L361, so server mode is affected identically.

It is already observable

The existing metric spikes under truncation, since it measures precisely this gap:

sampling/sampling_logp_difference/mean and .../max (grpo_trainer.py, _generate_and_score_completions)

A quick check: run GRPO+vLLM with top_p=1.0 and then top_p=0.8, and compare that metric. If it jumps with no other change, the bias is real. Worth doing before we pick a fix.

Prior art

• vLLM #49577 "Mask Replay" is merged (2026-08-13) but landed after v0.27.1 (2026-08-11), so it is not in any release covered by TRL's vllm>=0.18.0,<=0.27.1 pin. We cannot rely on it yet.
• slime shipped a training-side mask in v0.3.1.
• prime-rl #3235 "feat: top-p/top-k train sampling with sampling replay" is building the same thing independently.

Possible directions

1. Apply the same truncation to the training-side logits before the log-softmax, so both sides normalize over the same support. Mirrors what slime and prime-rl do, and keeps the temperature-symmetry precedent from #4159.
2. Ask vLLM for raw log-probs and apply temperature and truncation ourselves on both sides.
3. Wait for Mask Replay and lift the ceiling to vLLM 0.28. Doesn't help anyone on a supported version today.
4. Do nothing to the math, but warn when vllm_importance_sampling_correction=True is combined with truncation, so the bias is at least not silent.

I'd like a repro of the metric jump before committing to one.

Possibly related

• #4159
• #4772