WhisperLiveKit

WLK: Ultra-low-latency, self-hosted speech-to-text pipeline

### Powered by Leading Research: - Simul-[Whisper](https://arxiv.org/pdf/2406.10052)/[Streaming](https://arxiv.org/abs/2506.17077) (SOTA 2025) - Ultra-low latency transcription using [AlignAtt policy](https://arxiv.org/pdf/2305.11408). - [NLLW](https://github.com/QuentinFuxa/NoLanguageLeftWaiting) (2025), based on [distilled](https://huggingface.co/entai2965/nllb-200-distilled-600M-ctranslate2) [NLLB](https://arxiv.org/abs/2207.04672) (2022, 2024) - Simulatenous translation from & to 200 languages. - [WhisperStreaming](https://github.com/ufal/whisper_streaming) (SOTA 2023) - Low latency transcription using [LocalAgreement policy](https://www.isca-archive.org/interspeech_2020/liu20s_interspeech.pdf) - [Streaming Sortformer](https://arxiv.org/abs/2507.18446) (SOTA 2025) - Advanced real-time speaker diarization - [Qwen3-ASR-causal](https://github.com/QuentinFuxa/Qwen3-ASR-causal) (2026) - Causal streaming audio encoder for Qwen3-ASR: each audio block is encoded exactly once, constant compute per audio second, append-only transcripts. - [AlignAtt4LLM](https://github.com/QuentinFuxa/Alignatt4LLM) ([IWSLT 2026](https://arxiv.org/abs/2606.03967)) - Simultaneous translation with decoder-only LLMs: attention-gated commits, append-only output. > **Why not just run a simple Whisper model on every audio batch?** Whisper is designed for complete utterances, not real-time chunks. Processing small segments loses context, cuts off words mid-syllable, and produces poor transcription. WhisperLiveKit uses state-of-the-art simultaneous speech research for intelligent buffering and incremental processing. ### Architecture *The backend supports multiple concurrent users. Voice Activity Detection reduces overhead when no voice is detected.* ### Installation & Quick Start ```bash pip install whisperlivekit ``` #### Quick Start ```bash # Start the server — open http://localhost:8000 and start talking wlk --model base --language en # Auto-pull model and start server wlk run whisper:tiny # Transcribe a file (no server needed) wlk transcribe meeting.wav # Generate subtitles wlk transcribe --format srt podcast.mp3 -o podcast.srt # Manage models wlk models # See what's installed wlk pull large-v3 # Download a model wlk rm large-v3 # Delete a model # Benchmark speed and accuracy wlk bench ``` #### API Compatibility WhisperLiveKit exposes compatibility-oriented subsets of popular APIs: ```bash # OpenAI-compatible REST API curl http://localhost:8000/v1/audio/transcriptions -F [email protected] # Works with the OpenAI Python SDK client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") # Deepgram-compatible WebSocket # See docs/API.md for supported options and current SDK setup # Native WebSocket for real-time streaming ws://localhost:8000/asr ``` Per-session WebSocket query parameters: | param | example | effect | |---|---|---| | `language` | `?language=fr` | transcription language for this session (one shared engine serves mixed-language sessions) | | `target_language` | `?target_language=de` | translation target for this session (server must run with `--target-language`) | | `context` | `?context=WhisperLiveKit%2C+Qwen3-ASR` | terminology, names, or phrase-list text used to condition this session; supported by Whisper-family and SimulStreaming backends | | `mode` | `?mode=diff` | incremental snapshot/diff protocol instead of resending the full state (experimental, for integrators building their own client, see `diff_protocol.py`); the bundled web UI uses `full` | | `token` | `?token=...` | API token when the server runs with `--api-token` (also accepted as an `Authorization: Bearer` header) | See [docs/API.md](docs/API.md) for the complete API reference. For a native SwiftUI macOS client, see [macos/WhisperLiveKitMac](macos/WhisperLiveKitMac). > - See [here](whisperlivekit/whisper/tokenizer.py) for the list of all available languages. > - Check the [troubleshooting guide](docs/troubleshooting.md) for step-by-step fixes collected from recent GPU setup/env issues. > - For HTTPS requirements, see the **Parameters** section for SSL configuration options. #### Optional Dependencies | Feature | `uv sync` | `pip install -e` | |-----------|-------------|-------------| | **Apple Silicon MLX Whisper backend** | `uv sync --extra mlx-whisper` | `pip install -e ".[mlx-whisper]"` | | **FunASR SenseVoiceSmall** | `uv sync --extra funasr` | `pip install -e ".[funasr]"` | | **Voxtral (MLX backend, Apple Silicon)** | `uv sync --extra voxtral-mlx` | `pip install -e ".[voxtral-mlx]"` | | **CPU PyTorch stack** | `uv sync --extra cpu` | `pip install -e ".[cpu]"` | | **CUDA 12.9 PyTorch stack** | `uv sync --extra cu129` | `pip install -e ".[cu129]"` | | **Translation** | `uv sync --extra translation` | `pip install -e ".[translation]"` | | **Sentence tokenizer** | `uv sync --extra sentence_tokenizer` | `pip install -e ".[sentence_tokenizer]"` | | **Voxtral (HF backend)** | `uv sync --extra voxtral-hf` | `pip install -e ".[voxtral-hf]"` | | **Qwen3-ASR vLLM (CUDA)** | `uv sync --extra qwen3-vllm` | `pip install -e ".[qwen3-vllm]"` | | **Qwen3-ASR streaming (HF, CUDA/MPS/CPU)** | `uv sync --extra qwen3-streaming` | `pip install -e ".[qwen3-streaming]"` | | **Qwen3-ASR vLLM Metal (Apple Silicon)** | Install vLLM with the official vllm-metal script first, then `uv sync --extra qwen3-vllm-metal` | Install vLLM with the official vllm-metal script first, then `pip install -e ".[qwen3-vllm-metal]"` | | **Speaker diarization (Sortformer / NeMo 3)** | `uv sync --extra diarization-sortformer` | `pip install -e ".[diarization-sortformer]"` | | *[Not recommended]* Speaker diarization with Diart (Python 3.11 or 3.12) | `uv sync --extra diarization-diart` | `pip install -e ".[diarization-diart]"` | | **Canary-1b-v2 (NeMo, CUDA/CPU)** | `uv sync --extra canary` | `pip install -e ".[canary]"` | The Diart profile is limited to Python 3.11 and 3.12 because Diart 0.9.2 requires NumPy below 2. Use Sortformer for diarization on Python 3.13. Supported GPU profiles: ```bash # Profile A: Sortformer diarization uv sync --extra cu129 --extra diarization-sortformer # Profile B: Voxtral HF + translation uv sync --extra cu129 --extra voxtral-hf --extra translation # Profile C: Qwen3-ASR vLLM uv sync --extra qwen3-vllm ``` `qwen3-vllm` uses vLLM's CUDA wheel stack and must be installed in a separate environment from `cu129`. Several heavy extras (`voxtral-hf`, `qwen3-vllm-metal`, and the vLLM stacks) intentionally conflict with one another and must be installed in separate environments; the authoritative list is `[tool.uv].conflicts` in `pyproject.toml`. The `canary` extra conflicts with `voxtral-hf` and `qwen3-vllm-metal`, but is compatible with `diarization-sortformer` (both pull `nemo-toolkit[asr]`). See **Parameters & Configuration** below on how to use them.

Benchmarks use 6 minutes of public [LibriVox](https://librivox.org/) audiobook recordings per language (30s + 60s + 120s + 180s), with ground truth from [Project Gutenberg](https://www.gutenberg.org/). The figures above were measured on an NVIDIA H100 (CUDA); the qwen3 causal tower is English-only, so it only appears on the English chart. Fully reproducible with `python scripts/run_scatter_benchmark.py` (the script picks a matching combo set on Apple Silicon: mlx-whisper and voxtral-mlx instead of the CUDA-only backends). Raw results: [benchmarks/h100_scatter/](benchmarks/h100_scatter/). We are actively looking for benchmark results on other hardware (different NVIDIA GPUs, Apple Silicon chips, cloud instances). If you run the benchmarks on your machine, please share your results via an issue or PR! #### Use it to capture audio from web pages. Go to `chrome-extension` for instructions.

### Voxtral Backend WhisperLiveKit supports [Voxtral Mini](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602), a 4B-parameter speech model from Mistral AI that natively handles 100+ languages with automatic language detection. Whisper also supports auto-detection (`--language auto`), but Voxtral's per-chunk detection is more reliable and does not bias towards English. ```bash # Apple Silicon (native MLX, recommended) pip install -e ".[voxtral-mlx]" wlk --backend voxtral-mlx # Linux/GPU (HuggingFace transformers) pip install transformers torch wlk --backend voxtral ``` Voxtral uses its own streaming policy and does not use LocalAgreement or SimulStreaming. See [BENCHMARK.md](BENCHMARK.md) for performance numbers. ### FunASR / SenseVoiceSmall Install the optional backend and run [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) through WLK's existing LocalAgreement and VAC/VAD pipeline: ```bash pip install "whisperlivekit[funasr]" wlk --backend funasr --language auto ``` Use a verified local model snapshot without executing remote model code: ```bash wlk --backend funasr --model_dir /path/to/SenseVoiceSmall --language yue ``` The initial integration supports SenseVoiceSmall transcription in Mandarin (`zh`), Cantonese (`yue`), English (`en`), Japanese (`ja`), Korean (`ko`), and automatic detection. FunASR uses LocalAgreement only; selecting it with the default policy switches that policy automatically. It does not support `--direct-english-translation`, and WLK remains responsible for voice activity control rather than enabling FunASR's internal VAD. This compatibility contract does not cover arbitrary FunASR models. SenseVoiceSmall is distributed under its [model license](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE). ### Qwen3-ASR streaming (HF Transformers) `qwen3-streaming` runs Qwen3-ASR through plain HF Transformers with a bounded-recompute audio cache: the pretrained audio tower only re-encodes a local window (default 12 s) per update, cached audio embeddings are append-only, and text is committed with a stable-prefix rule. Works on CUDA, Apple Silicon (MPS) and CPU, no vLLM required. ```bash pip install -e ".[qwen3-streaming]" wlk --backend qwen3-streaming --language en ``` Notes: - An explicit `--language` is required (automatic detection switches language mid-stream on accented audio). - Word timestamps are interpolated estimates (~1 s precision): good enough for lines and diarization alignment. Use `--backend qwen3-vllm` (ForcedAligner) when exact word timing matters. - Decode pacing self-adjusts to the hardware; on GPUs slower than real time the update cadence grows instead of lagging. Plan one realtime session per GPU. - Defaults encode the validated operating point (12 s left context, ~15 s segments); see `--help` for the `--qwen3-streaming-*` knobs. **Causal mode (minimum compute per chunk).** The windowed default re-encodes up to 12 s of audio on every update. The causal mode runs an append-only causal-KV encoder instead: each ~2 s audio block is encoded exactly once, memory is bounded (15 s window + sentence-boundary segment resets), and per-chunk compute is constant in stream length: ```bash wlk --backend qwen3-streaming --language en \ --qwen3-streaming-audio-backend causal \ --qwen3-streaming-tower-checkpoint qfuxa/qwen3-asr-0.6b-st