multilingual=True silently returns translated text unless condition_on_previous_text=False

Summary

With multilingual=True, a 30-second window whose language is detected as English with p = 1.0 is still decoded in the previous window's language. No error, no warning — just plausible text in the wrong language, which is the expensive kind of failure: it looks like a result.

The cause is that previous_tokens (the condition_on_previous_text prompt) is bound before the per-window language switch and is never reset when the language changes, so the previous language's text travels along as <|startofprev|> context and outweighs the freshly set <|en|> token.

Setting condition_on_previous_text=False works around it. Since that parameter defaults to True, the default configuration of multilingual=True does not produce a multilingual transcript.

Environment

• faster-whisper 1.2.1 (current release; also verified against master @ ed9a06cd, the code in question is unchanged)
• model large-v3, device="cuda", compute_type="float16"
• Windows 11, CTranslate2 via pip

Reproduction

Audio: a single 16 kHz mono WAV, 161.9 s, built from text-to-speech so it can be shared freely:

The language switch sits almost exactly on the 90 s window boundary, so window 4 (90–120 s) is essentially pure English preceded by three full German windows.

import faster_whisper

class LogProxy:
    """Logs every detect_language answer and passes it through UNCHANGED."""
    def __init__(self, real):
        self._real, self.log = real, []
    def detect_language(self, enc):
        r = self._real.detect_language(enc)
        tok, p = r[0][0]
        self.log.append((tok[2:-2], round(float(p), 3)))
        return r
    def __getattr__(self, name):
        return getattr(self._real, name)

m = faster_whisper.WhisperModel("large-v3", device="cuda", compute_type="float16")

def run(**kw):
    proxy = LogProxy(m.model)
    m.model = proxy
    try:
        segs, _ = m.transcribe("de_en_de.wav", language="de", beam_size=5,
                               vad_filter=False, **kw)
        segs = list(segs)
    finally:
        m.model = proxy._real
    print(proxy.log)
    for s in segs:
        print(f"[{s.start:6.2f} -> {s.end:6.2f}] {s.text.strip()}")

run(multilingual=True)                                      # (b)
run(multilingual=True, condition_on_previous_text=False)    # (c)

Result

Both runs detect the same languages per window — window 4 is en at p = 1.0:

(b) [('de', 0.999), ('de', 1.0), ('de', 0.999), ('en', 1.0), ('de', 0.653), ('de', 0.993)]
(c) [('de', 0.999), ('de', 1.0), ('de', 0.999), ('en', 1.0), ('en', 0.513), ('de', 0.995)]

The first segment of window 4 differs completely:

The spoken sentence is "I came here from Manchester with my club about three years ago." So with multilingual=True at the default settings the output is indistinguishable from the multilingual=False reference — the feature has no effect other than degrading the text.

Cause

faster_whisper/transcribe.py, _generate_segments (line numbers from 1.2.1 / current master):

1187:  previous_tokens = all_tokens[prompt_reset_since:]   # bound BEFORE the switch
...
1192:  if options.multilingual:
1193:      results = self.model.detect_language(encoder_output)
1194:      language_token, language_probability = results[0][0]
1195:      language = language_token[2:-2]
1196:
1197:      tokenizer.language = tokenizer.tokenizer.token_to_id(language_token)
1198:      tokenizer.language_code = language
1199:                                                       # prompt_reset_since untouched
1200:  prompt = self.get_prompt(
1201:      tokenizer,
1202:      previous_tokens,                                 # carries the old language
...

Note that resetting prompt_reset_since inside the if block is not sufficient on its own: previous_tokens is already bound at line 1187, so the change would only take effect from the next window onwards — the first window of a foreign-language passage would still be translated.

Suggested fix (verified)

Re-bind previous_tokens on an actual language change:

 if options.multilingual:
     results = self.model.detect_language(encoder_output)
     language_token, language_probability = results[0][0]
     language = language_token[2:-2]

+    if language != tokenizer.language_code:
+        prompt_reset_since = len(all_tokens)
+        previous_tokens = all_tokens[prompt_reset_since:]
+
     tokenizer.language = tokenizer.tokenizer.token_to_id(language_token)
     tokenizer.language_code = language

With this patch and condition_on_previous_text left at its default, the same run returns:

[ 90.00 ->  93.54] I came here from Manchester with my club about three years ago.
[ 94.42 -> 100.06] The first winter was quite a shock, honestly, because back home we never get snow like that.
[100.96 -> 105.88] My name is Peter Whitfield and I work as an engineer for a small company near the railway station.

and the switch back to German at 139.2 s still works.

The comparison against the previous language matters: without it the prompt would be cleared at every window, which would throw away condition_on_previous_text entirely.

What this fix does not solve

Windows that contain a language boundary remain wrong, for a different reason: the per-window detection result is taken unconditionally at line 1194, and language_probability is assigned there but never used. language_detection_threshold is applied only inside detect_language (line 1837), not to the per-window detection.

In the patched run, window 5 (120–150 s, English plus the start of the German closing) was detected as en with p = 0.513 and about 8 s of German went missing. So a complete fix probably needs a confidence threshold on the per-window detection as well. That half may be the same underlying cause as #1322 — I have posted a measurement there rather than guessing here.

The 3.6 MB zipped WAV can be attached on request; the effect should reproduce with any German-then-English recording where the switch falls near a 30-second boundary.