Streaming: when the upstream closes the stream mid-answer, the gateway ends its own response cleanly (200, no error event, no `[DONE]`) — the client cannot tell the answer is incomplete

What happens

If the upstream provider drops a streaming response part-way through, the gateway forwards the tokens it received and then closes its own response as if everything were fine: HTTP 200, no data: [DONE], no chunk with a finish_reason, no data: {"error": ...} event. The configured fallback target is never tried.

From the application's side this looks like a short but complete answer: it shows or stores half an answer, and nothing is logged, because nothing failed.

I hit this with a fallback strategy configured (two targets, strategy.mode: fallback), so the gateway had somewhere to go.

How to reproduce

One fake OpenAI-compatible provider, one command. The provider's model name selects the fault: fb-s08-stream-cut sends 40 of 60 tokens and then closes the connection without a terminating chunk.

git clone https://github.com/failoverbench/failoverbench && cd failoverbench && pip install -e .
python -m failoverbench wall --host 0.0.0.0 --profile full &      # the fake provider on :8401
docker compose -f gateways/portkey/docker-compose.yml up -d       # portkeyai/gateway:latest on :8787
python -m failoverbench run --system systems/portkey-gateway.yaml --profile full --only S08,S09,S14

Or by hand — one streaming chat completion to http://127.0.0.1:8787/v1/chat/completions with model fb-s08-stream-cut and this x-portkey-config:

{"strategy":{"mode":"fallback"},"request_timeout":30000,
 "targets":[
  {"provider":"openai","api_key":"unused","custom_host":"http://host.docker.internal:8401/v1",
   "override_params":{"model":"fb-s08-stream-cut"},
   "retry":{"attempts":2,"on_status_codes":[429,500,502,503,504],"use_retry_after_header":true}},
  {"provider":"openai","api_key":"unused","custom_host":"http://host.docker.internal:8401/v1",
   "override_params":{"model":"fb-ok"}}]}

(The fake provider ignores API keys.)

What I observed

• Gateway → client: 200, text/event-stream, 40 content chunks, then the chunked body ends (0\r\n\r\n). No [DONE], no finish_reason, no error event.
• Fake provider's log: one request to fb-s08-stream-cut, zero requests to fb-ok (the fallback target).
• Same result when the upstream stalls for 30 s after 20 tokens and then closes (fb-s09-stream-stall): 20 tokens delivered, clean close at 30.3 s, no error, no fallback.
• Same result in the prefill-fallback case (fb-s14-stream-cut).

What I expected

Either would be fine:

1. try the fallback target and deliver a complete answer, or
2. report the failure in-band — data: {"error": {...}} before closing — so the client can tell a cut stream from a finished one.

Closing cleanly is the hardest case for a client: the openai SDK and most clients accept a stream that ends without a finish_reason, so the truncation goes unnoticed. For comparison, the same fake provider through LiteLLM (Router and proxy), Bifrost, LangChain and the openai SDK surfaces the cut as an error to the caller (the LiteLLM proxy and Bifrost use an in-band error event).

Environment

portkeyai/gateway:latest pulled 8 Sep 2026 (portkeyai/gateway@sha256:97f094d9c8a764cbfaa2a7138c0017b247ca923bb06db1b4c13b7f8a33b5200d), Docker Desktop on macOS; upstream reached with custom_host over host.docker.internal.

How I found this: Failover Bench is an open harness — a fake provider that misbehaves in sixteen documented ways, run against gateways and SDKs with the same baseline settings. Results are published at https://failoverbench.github.io/failoverbench/ and updated on the first Monday of each month (next: 5 Oct 2026). Happy to re-run against a fix or a branch before then, and to adjust the harness if any of the above is a harness mistake. Not affiliated with any vendor.