POST /v1/audio/transcriptions: ndjson streaming reports inference-time failures as 200 with an empty body

Edited 2026-08-07. The original report blamed a missing model-load guard and framed this as #724 left unfixed on this endpoint. That was wrong — _preflight_model_load is already present here. The real mechanism is below; the original text and my correction are preserved in the comments. Fix proposed in #877.

Summary

POST /v1/audio/transcriptions returns HTTP 200 with a zero-byte body when transcription fails during inference. The client sees a fast, successful, empty transcription that never happened.

Repro

mlx-community/whisper-tiny reproduces it for 70MB (its snapshot ships no processor files):

curl -s -X POST http://127.0.0.1:8899/v1/audio/transcriptions \
  -F "[email protected]" -F "model=mlx-community/whisper-tiny" \
  -w "\nHTTP=%{http_code}\n"

Observed: HTTP=200, body length 0, ~8ms. Server log:

File ".../mlx_audio/stt/models/whisper/whisper.py", line 786, in _detect_language
File ".../mlx_audio/stt/models/whisper/decoding.py", line 36, in detect_language
File ".../mlx_audio/stt/models/whisper/whisper.py", line 741, in get_tokenizer
ValueError: Processor not found. Make sure the model was loaded with a HuggingFace processor.
INFO: 127.0.0.1:59499 - "POST /v1/audio/transcriptions HTTP/1.1" 200 OK

Note the last line: the exception is raised and the response is still 200 OK.

Mechanism

Not a missing preflight. stt_transcriptions already calls await _preflight_model_load(payload.model) and it succeeds — the model loads fine.

1. Whisper fails later, during inference, in get_tokenizer(). A load-time preflight cannot catch that.
2. The default response_format is ndjson, which returns StreamingResponse(_stream_inference_results(...)). Status and headers are committed before the generator body runs.
3. _stream_inference_results does raise chunk.error inside that already-committed generator. The status can no longer change, so the stream just ends empty.

The non-streaming formats are already correct: text / json / verbose_json accumulate and raise chunk.error before responding. The bug is specific to the streaming ndjson path — which is the default, so it is what most callers hit.

Impact

We hit this while benchmarking STT throughput across 1800 requests. Had the harness trusted the status code, whisper would have been recorded as 100% successful at ~8ms/request — an aggregate throughput roughly two orders of magnitude too high, and the fastest model in the comparison. A crash would have been harmless by comparison; this silently produces confident, publishable, fabricated numbers.

Proposed fix — #877

Emit a terminal ndjson error record instead of raising into a committed stream, gated behind a keyword-only flag defaulting to off so the TTS audio/* path is untouched (a JSON line there would corrupt the audio body).

Verified live on Apple silicon: the bug case now returns the actual Processor not found / ValueError, parakeet transcription is unchanged, and /v1/audio/speech still returns valid binary audio.

The status necessarily stays 200 once headers are committed — what changes is that the body says what went wrong instead of being silently empty.

Environment

mlx-audio 0.4.7 (latest at time of filing), macOS 26.5.2, Apple silicon.

Workaround for clients

Do not trust the status code on this endpoint. Validate that the body parses as JSON and contains a string text field; treat an empty body as a failure.