ExponentialDecayLengthPenalty creates NaN EOS scores when combined with minimum-length constraints

System Info

• Transformers: 5.19.0.dev0, main commit d6c1e71bd717bf092f8293f0c3c9bd4a5ac5401a; also reproduced on stable 5.18.0.
• Python: 3.12.14
• PyTorch: 2.14.1+cpu
• Accelerate: 1.15.0
• Platform: Linux x86-64, CPU, single process.
• Tiny models constructed from configuration; no model downloads or custom logits processors.

Who can help?

@Cyrilvallez

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

When exponential_decay_length_penalty starts before min_new_tokens is reached, it turns the masked EOS score into NaN. Greedy and beam decoding can terminate before the minimum; sampling raises an error.

import torch
import transformers
from transformers import GPT2Config, GPT2LMHeadModel
from transformers.generation.logits_process import (
    ExponentialDecayLengthPenalty,
    MinNewTokensLengthLogitsProcessor,
)

torch.set_num_threads(1)
torch.manual_seed(123)
print("Transformers:", transformers.__version__)

# Isolate the processor interaction: valid non-EOS tokens remain available.
ids = torch.tensor([[1, 3]])
scores = torch.arange(6, dtype=torch.float32).unsqueeze(0)
masked = MinNewTokensLengthLogitsProcessor(1, 3, 2)(ids, scores)
processed = ExponentialDecayLengthPenalty((0, 1.1), 2, 1)(ids, masked)
print("EOS before/after decay:", masked[0, 2].item(), processed[0, 2].item())

# Reproduce through the public generate() API.
model = GPT2LMHeadModel(GPT2Config(
    vocab_size=8, n_positions=16, n_embd=8, n_layer=1, n_head=2,
    bos_token_id=1, eos_token_id=2, pad_token_id=0,
)).eval()

for mode in ("greedy", "beam", "sample"):
    try:
        output = model.generate(
            torch.tensor([[1]]),
            min_new_tokens=3,
            max_new_tokens=5,
            exponential_decay_length_penalty=(0, 1.1),
            do_sample=(mode == "sample"),
            num_beams=2 if mode == "beam" else 1,
            return_dict_in_generate=True,
            output_scores=True,
        )
        print(mode, output.sequences.tolist())
        print("NaN per step:", [s.isnan().any().item() for s in output.scores])
    except RuntimeError as error:
        print(mode, type(error).__name__, str(error))

Observed on the pinned main revision and stable 5.18.0:

EOS before/after decay: -inf nan
greedy [[1, 1, 2]]
beam [[1, 1, 2]]
sample RuntimeError probability tensor contains either `inf`, `nan` or element < 0

The greedy/beam outputs contain only two new tokens, including EOS, despite min_new_tokens=3. The individual minimum-only and decay-only configurations do not produce NaNs. The combined failure also reproduces with a tiny T5 model and with min_length.

Expected behavior

EOS should remain masked until the requested minimum length is reached. A soft EOS preference should preserve that mask and should not introduce NaNs or cause sampling to fail. This configuration has valid non-EOS continuations throughout the minimum-length interval.

The minimum-length options specify a minimum generation length. ExponentialDecayLengthPenalty encourages shorter outputs by increasing EOS scores.

Suspected cause and possible fix

The additive boost computes abs(eos_score) * (factor ** step - 1). For a masked EOS and a factor greater than 1, adding that boost computes -inf + inf, producing NaN. A factor of 1 also produces NaN through inf * 0.

A local correction applies the boost only to finite EOS scores, preserving existing -inf masks:

eos_scores = scores[:, self.eos_token_id]
penalty = torch.where(
    torch.isfinite(eos_scores),
    torch.abs(eos_scores) * (pow(self.regulation_factor, penalty_idx) - 1),
    0.0,
)

This replaces the existing one-line penalty calculation. The remaining update can stay as it is.

Validation and related work

• 21 focused cases covering masked/finite EOS scores, multiple EOS IDs, float32/float16/bfloat16, and GPT-2/T5 under greedy, sampling and beam decoding fail before the correction and pass afterward.
• All 48 existing tests in tests/generation/test_logits_process.py pass with this correction and a separate encoder-penalty correction applied.
• The additive formula was introduced by #25594 to fix finite negative EOS scores. This report concerns masked infinite scores, rather than that finite-score behavior.
• Searches of issues and PRs did not find an equivalent masked-EOS/minimum-length report or active fix.

I can submit a PR with a focused fix and regression tests if this approach looks appropriate.

AI assistance was used to investigate this behavior and prepare this report. The reproduction and local correction were executed and verified in the environments above.