VibeVoice-ASR: float32 ceil in get_audio_features miscounts audio tokens for long audio, crashing generate() with a mask shape mismatch

System Info

• transformers version: 5.6.2
• Platform: Linux (WSL2), Python 3.11.15
• PyTorch version (GPU?): 2.11.0+cu130 (True), CUDA 13.0
• Model: microsoft/VibeVoice-ASR-HF (also reproduced with an NF4-quantized copy of the same architecture)

As of 2026-09-15 the divergence is still present on main: the processor counts placeholders with np.ceil (float64) while get_audio_features recomputes the count with torch.ceil on a float32 quotient, so this affects both v5.6.2 and current main.

Who can help?

@eustlb @ebezzam (audio models)

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Transcribing a single long audio input (~21 minutes) with VibeVoiceAsrForConditionalGeneration.generate() fails with:

RuntimeError: The shape of the mask [1, 9476] at index 1 does not match the shape of the indexed tensor [1, 9477, 3584] at index 1

Root cause: the processor and the model compute the number of audio placeholder tokens with different floating-point precisions, and they disagree for long inputs.

VibeVoiceAsrProcessor.__call__ counts placeholders in float64 (processing_vibevoice_asr.py, line 157): num_audio_tokens = np.ceil(audio_lengths / audio_kwargs["pad_to_multiple_of"]).astype(int).

VibeVoiceAsrModel.get_audio_features recomputes the same quantity in float32 (modeling_vibevoice_asr.py, line 356; same code in modular_vibevoice_asr.py, line 242): num_audio_tokens = torch.ceil(padding_mask.sum(dim=-1) / self.config.acoustic_tokenizer_encoder_config.hop_length).to(torch.int64). Here padding_mask.sum(dim=-1) is int64, so true division promotes to the default dtype float32.

float32 has a 24-bit significand, so above 2**24 samples (about 11.65 minutes at 24 kHz) not every sample count is representable. When the sample count lies just above a multiple of the 3200-sample hop, the float32 quotient rounds down to the exact hop boundary and torch.ceil returns one less than np.ceil in float64. The processor then emits N+1 placeholder tokens and padded features while the model builds a mask for N tokens, and combined_features[padding_mask] raises the shape mismatch above.

Minimal demonstration of the disagreement (no model weights needed):

import numpy as np
import torch

hop = 3200
samples = 9476 * hop + 1  # 30_323_201 samples at 24 kHz ~= 21.06 min
processor_count = int(np.ceil(samples / hop))                                # 9477
model_count = int(torch.ceil(torch.tensor([samples], dtype=torch.int64) / hop))  # 9476
print(processor_count, model_count)  # 9477 9476 -> mismatch

Full reproduction with the model (synthetic silence is enough; any audio whose 24 kHz sample count is k * 3200 + 1 with k * 3200 + 1 > 2**24 triggers it):

import numpy as np
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration

model_id = "microsoft/VibeVoice-ASR-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto", dtype="auto")

audio = np.zeros(9476 * 3200 + 1, dtype=np.float32)  # interpreted at 24 kHz
inputs = processor.apply_transcription_request(audio=audio).to(model.device, model.dtype)
out = model.generate(**inputs)
# RuntimeError: The shape of the mask [1, 9476] at index 1 does not match the shape of the indexed tensor [1, 9477, 3584] at index 1

Related issues (checked, but different bugs)

• #47672 / #47673 (merged): fixes a dtype mismatch in the masked_scatter call of the same code path. This is a different bug: the crash reported here happens earlier, because processor and model disagree about how many audio tokens exist at all, so the dtype fix does not address it.
• #46472 (merged): uses the same ceil(samples / 3200) arithmetic for the vLLM max_source_positions budget, confirming the hop-length relationship, but does not touch the float32/float64 divergence between processor and model.

Expected behavior

generate() should transcribe the audio. The token count in get_audio_features should be computed exactly, e.g. with integer ceil division instead of float division:

hop = self.config.acoustic_tokenizer_encoder_config.hop_length
num_audio_tokens = torch.div(padding_mask.sum(dim=-1) + hop - 1, hop, rounding_mode="floor")

This matches the processor's float64 result for all input lengths and avoids the float32 precision cliff entirely. The fix would need to go into modular_vibevoice_asr.py (line 242) so the generated modeling_vibevoice_asr.py picks it up.