#407 Speaker diarization collapses on tiny waveform changes (e.g. +0.7 dB linear gain)

Summary

VibeVoice-ASR speaker diarization is extremely sensitive to any modification of the input waveform, even when the modification is perceptually identical to human ears. The same 22.85s vocal-only clip processed in 5 different ways — all sounding identical — produces wildly different speaker counts, often collapsing 3 speakers into 1.

This makes the model very hard to deploy reliably in any pipeline that performs even minor preprocessing.

Reproduction

A 22.85s vocal-only clip (mono, 44.1 kHz PCM, after vocal separation) is sent to the same VibeVoice-ASR endpoint with identical parameters (do_sample=False, num_beams=1, temperature=0). The only thing varied is how the audio is preprocessed before being sent. All variants are perceptually indistinguishable.

Variant	RMS (dBFS)	Crest factor	Speakers detected
Original (no preprocessing)	−25.72	17.51 dB	3 ✅ (correct ground truth)
+0.72 dB linear gain only	−25.00	17.51 dB	2 ❌
Linear gain to −18 LUFS	−22.20	17.51 dB	1 ❌❌
ffmpeg `loudnorm` to −18 LUFS	−22.25	18.51 dB	2 ❌
ffmpeg `loudnorm` to −23 LUFS	−27.25	18.51 dB	1 ❌❌

A 0.72 dB pure linear gain (well below the threshold of human audibility) is enough to make the model lose 1 of 3 speakers.

I confirmed the model server is fully deterministic by calling the same file 5 times in a row — produces byte-identical results. So the variance above is not model randomness; it is genuinely caused by the waveform modifications.

Root Cause Analysis

I traced this through vibevoice/processor/audio_utils.py::AudioNormalizer:

def tailor_dB_FS(self, audio):
    rms = np.sqrt(np.mean(audio ** 2))
    scalar = 10 ** (self.target_dB_FS / 20) / (rms + self.eps)
    return audio * scalar, rms, scalar

def avoid_clipping(self, audio, scalar=None):
    if scalar is None:
        max_val = np.max(np.abs(audio))
        if max_val > 1.0:                  # <-- step discontinuity
            scalar = max_val + self.eps
        else:
            scalar = 1.0
    return audio / scalar, scalar

Two issues amplify input sensitivity:

Step discontinuity in avoid_clipping: the if max_val > 1.0 branch creates a non-continuous transformation. Inputs whose post-tailor_dB_FS peaks straddle 1.0 get scaled differently. Different external preprocessing changes whether the threshold is hit, leading to different effective input levels reaching the LLM.
Full-segment RMS includes silence: integrating silence frames into the RMS estimate makes the normalization sensitive to how much non-speech is at the boundaries of the clip.

But these only amplify the problem. The deeper issue appears to be that the LLM-based generation produces speaker IDs as autoregressive tokens — when the logits for <speaker_0> vs <speaker_1> are close, even tiny floating-point input perturbations flip the decision. Once flipped, autoregressive generation propagates the error to all subsequent sentences (which is why we see the "collapse to 1 speaker" failure mode).

Why This Matters

Routine preprocessing operations that any production pipeline does, will break diarization:

Audio loudness normalization (very common in podcast / video pipelines)
Vocal separation (output sample rate / format may vary by tool)
MP3 / Opus / AAC re-encoding at any bitrate
Sample rate conversion (44.1k ↔ 48k ↔ 24k)
Even just a +1 dB gain to help upstream VAD detect quieter speech

Users will see non-deterministic diarization quality and have no way to debug it because all the audio sounds identical to them.

Suggestions

Document the problem: add a section to docs/vibevoice-asr.md warning users that any preprocessing of the input can degrade speaker diarization. Recommend feeding raw audio with no normalization or re-encoding.
Make AudioNormalizer more stable:
- Replace the hard if max_val > 1.0 step with a soft limiter (e.g. tanh-based or a small linear ramp at 0.95–1.0) to remove the discontinuity.
- Use VAD-gated RMS (only accumulate frames above a noise floor) to make the level estimate robust to silence padding.
Add training-time augmentation: random gain (±6 dB), MP3 / Opus compression at multiple bitrates, and resampling are all standard for ASR / SV training (e.g. Whisper, ECAPA-TDNN). VibeVoice's behavior strongly suggests it was trained without such augmentation.
Offer a "speaker-stable" inference mode: optionally process the input through a deterministic canonical pipeline (e.g. always resample to 24 kHz mono, full-segment RMS to a fixed dBFS, no clip avoidance) so users can rely on consistent results regardless of upstream preprocessing.

Environment

Model: microsoft/VibeVoice-ASR (HF endpoint, gradio API)
Inference flags: do_sample=False, num_beams=1, temperature=0, repetition_penalty=1.0
Audio: 22.85s, mono, vocal-separated, originally 44.1 kHz / 16-bit PCM
Test conducted on 2026-06-14

I have all 5 variant audio files + per-variant JSON outputs available; happy to attach them as a comment if useful.

Related Issues

#257 (inconsistent prompt tokens / failed transcripts — may share root cause)
#256 (VIBEVOICE_USE_MEAN for determinism in vLLM but not HF — suggests team is already aware of input sensitivity)
#373 ("Yes. Yes." infinite loop — another fragility symptom)

Thanks for the great work. Looking forward to feedback on whether this brittleness is being addressed.