How to speed up diarization speed for WhisperX?
Author: yccheokCreated Nov 4, 2025Updated Sep 12, 2026
I am currently encountering a diarization-speed issue when using WhisperX.
Based on https://github.com/m-bain/whisperX/issues/499 , the likely cause is that diarization is being executed on the CPU rather than the GPU.
I have already attempted the workaround mentioned there. Below is my Dockerfile — I’m running this on RunPod — for your reference.
FROM runpod/pytorch:cuda12
# Set the working directory in the container
WORKDIR /app
# Install ffmpeg, vim
RUN apt-get update && \
apt-get install -y ffmpeg vim
# Install WhisperX via pip
RUN pip install --upgrade pip && \
pip install --no-cache-dir runpod==1.7.7 whisperx==3.3.1 pyannote.audio==3.3.2 torchaudio==2.8.0 matplotlib==3.10.7
# https://github.com/m-bain/whisperX/issues/499
RUN pip uninstall -y onnxruntime && \
pip install --force-reinstall --no-cache-dir onnxruntime-gpu
# Download large-v3 model
RUN python -c "import whisperx; whisperx.load_model('large-v3', device='cpu', compute_type='int8')"
# Initialize diarization pipeline
RUN python -c "import whisperx; whisperx.DiarizationPipeline(use_auth_token='xxx', device='cpu')"
# Copy source code into image
COPY src src
# -u disables output buffering so logs appear in real-time.
CMD [ "python", "-u", "src/handler.py" ]This is my Python code.
import runpod
import whisperx
import time
start_time = time.time()
diarize_model = whisperx.DiarizationPipeline(
use_auth_token='...',
device='cuda'
)
end_time = time.time()
time_s = (end_time - start_time)
print(f" whisperx.DiarizationPipeline done: {time_s:.2f} s")For a one-minute transcription, it also takes about one minute to perform diarization, which feels quite slow to me.
diarize_segments = diarize_model(audio)I was wondering what else I could try to speed up the diarization process.
Thank you.
Source: m-bain/whisperX