Feature Request: Add NVIDIA NeMo Sortformer as an Optional Speaker Diarization Backend

Summary

Would you consider adding support for NVIDIA NeMo Sortformer as an optional speaker diarization backend in WhisperX?

In particular, NVIDIA currently provides the following streaming diarization model:

nvidia/diar_streaming_sortformer_4spk-v2.1

WhisperX already has a strong pipeline for ASR and word-level alignment, while NeMo Sortformer is focused specifically on speaker diarization. The two systems appear to be complementary and could potentially be integrated by converting Sortformer diarization output into the format expected by assign_word_speakers().

Motivation

WhisperX currently uses pyannote for speaker diarization.

It would be useful to support NeMo Sortformer as an alternative backend for several reasons:

• Sortformer is an end-to-end speaker diarization model from NVIDIA NeMo.
• The newer streaming Sortformer models are specifically optimized for meeting speech.
• It may perform differently or better than pyannote on certain datasets, especially meetings and conversational audio.
• It would allow users to benchmark multiple diarization backends while keeping the same WhisperX transcription and alignment pipeline.
• The streaming Sortformer models could also be useful for future real-time or low-latency WhisperX-related workflows.

Possible Architecture

A possible pipeline could look like this:

Audio
 ├── WhisperX
 │    ├── faster-whisper transcription
 │    └── forced alignment
 │          ↓
 │      word timestamps
 │
 └── NVIDIA NeMo Sortformer
           ↓
      speaker segments

word timestamps + speaker segments
           ↓
assign_word_speakers()
           ↓
speaker-attributed transcript

Ideally, NeMo would only replace the diarization stage, while WhisperX would continue handling transcription and word-level alignment.

For example, the NeMo output could be converted into a WhisperX-compatible structure:

diarize_segments = pd.DataFrame([
    {
        "start": 0.32,
        "end": 4.80,
        "speaker": "SPEAKER_00",
    },
    {
        "start": 4.76,
        "end": 9.21,
        "speaker": "SPEAKER_01",
    },
])

result = whisperx.assign_word_speakers(
    diarize_segments,
    result,
)

This could potentially be implemented as a separate diarization backend or adapter without changing the WhisperX ASR/alignment pipeline.

Model

The model I am particularly interested in is:

nvidia/diar_streaming_sortformer_4spk-v2.1

It supports streaming diarization for up to four speakers and has been improved for meeting speech.

An offline Sortformer backend could also be useful if that fits WhisperX's architecture better.

Questions

1. Would the maintainers be interested in supporting NeMo Sortformer as an optional diarization backend?
2. Is assign_word_speakers() considered the recommended integration point for an external diarization system?
3. Would a PR implementing a NeMo/Sortformer adapter be welcome?
4. Would you prefer this to live inside WhisperX, or as an external plugin/integration?

I think keeping pyannote as the default while allowing NeMo Sortformer as an optional backend could provide users with more flexibility without changing the existing workflow.