STT handlers retain the detected language across sessions; only Parakeet resets it

Author: HotragnCreated Sep 8, 2026Updated Sep 8, 2026

Summary

In --language auto, four of the five bundled STT handlers keep self.last_language when a session ends. Pipeline units are reused across clients, so the next client inherits the previous client's detected language.

ParakeetTDTSTTHandler — the default backend — already resets it, which is what makes this look like an oversight rather than a policy:

python
# STT/parakeet_tdt_handler.py
def on_session_end(self) -> None:
    super().on_session_end()
    self.last_language = self.start_language if self.start_language else "en"

BaseSTTHandler.on_session_end only clears revision bookkeeping:

python
def on_session_end(self) -> None:
    if hasattr(self, "_completed_final_revision_keys"):
        self._completed_final_revision_keys.clear()
handler resets last_language
parakeet_tdt_handler (default) yes
whisper_stt_handler no
mlx_audio_whisper_handler no
lightning_whisper_mlx_handler no
qwen3_asr_handler no

on_session_end is the documented place for this — websocket_router._clean_unit describes SESSION_END as "the soft reset signal for stateful handlers", and flushes the queues specifically so work "cannot be picked up by handlers and leak into the next session that claims this unit."

Impact, per backend

I want to be precise rather than overstate this, because it varies:

Qwen3-ASR — user-visible. Progressive requests are forced into last_language:

python
# STT/qwen3_asr_handler.py:148
request_language = self.forced_language
if request_language is None and progressive:
    request_language = self.last_language

So with --enable_live_transcription, a new client's live partials are transcribed in the previous client's language until their first final turn re-detects. The handler's own docstring says progressive windows reuse "the language of the last final turn" — across a session boundary that's the wrong conversation's last final turn.

Whisper family — fallback only. last_language is consulted when detection yields nothing, so the leak surfaces on the new session's first turn only if detection is unavailable or returns an unsupported code. Milder. Note process() copies gen_kwargs (dict(self.gen_kwargs)), so the forced-language kwarg is not mutated at runtime — I checked, and only last_language persists.

Reproduction

No model needed; this is handler state, and it is what SESSION_END is supposed to clear:

python
from speech_to_speech.STT.whisper_stt_handler import WhisperSTTHandler
from speech_to_speech.STT.parakeet_tdt_handler import ParakeetTDTSTTHandler

w = object.__new__(WhisperSTTHandler)
w.start_language = "auto"; w.last_language = "de"; w.gen_kwargs = {}
w.on_session_end()
print(w.last_language)   # 'de'  <- survives into the next session

p = object.__new__(ParakeetTDTSTTHandler)
p.start_language = "auto"; p.last_language = "de"; p.enable_live_transcription = False
p.on_session_end()
print(p.last_language)   # 'auto' <- reset

End to end: run serve with --stt qwen3-asr --language auto --enable_live_transcription, connect a German speaker, disconnect, then connect a French speaker to the same pipeline unit. The second client's live partials come back German-forced until their first final turn.

Suggested fix

Reset the per-session language state where the convention already lives. Either:

  1. Give each handler an on_session_end matching Parakeet's, or
  2. Do it once in BaseSTTHandler.on_session_endstart_language/last_language is a shared convention across all five handlers, so one reset covers every current backend and any future one, which is the part I'd value most given this was missed four times.

Happy to open a PR; I'd default to (2) with the contract documented on the base class, unless you'd rather keep it explicit per handler.

One adjacent thing I noticed while checking, not part of this: Parakeet resets to self.start_language, which is the literal string "auto" in auto mode rather than a language code. That's a separate question from whether the reset happens.

Environment

  • speech-to-speech v1.0.0, main at 16d7f98
  • Affects the realtime server, where PipelineUnits are claimed and released across clients

Source: huggingface/speech-to-speech