#60479·istio

503UC returned when connection is terminated before body stream completes

Author: StonoCreated Jun 5, 2026Updated Sep 17, 2026

Labelsarea/networkinglifecycle/stale

Is this the right place to submit this?

This is not a security vulnerability or a crashing bug
This is not a question about how to use Istio

Bug Description

Hello, I believe I've managed to recreate a 503UC issue.

Given the following setup:

client(curl) -> [ingress load balancer -> outbound sidecar] -----> [inbound sidecar -> application]

bash

curl -i -X POST https://myapp/upload \
  -F "[email protected]"

In my example, client is streaming a multipart request of 3mb. application is reading the request headers (before consuming the body) and rejecting the request.

Locally, this returns a 401 as expected. However when deployed (and subsequently fronted with istio) we intermittently get 503s.

I ran ksniff on the cluster, and can clearly see the same behaviour we see locally, which is the application returning the 401 to the proxy and terminating the connection:

However the response we receive is a 503 from envoy:

This is confirmed in the inbound-sidecar logs too:

{"time":"2026-06-05T11:32:08.443369Z","level":"debug","scope":"envoy misc","msg":"Unknown error code 32 details Broken pipe","caller":"external/envoy/source/common/network/io_socket_error_impl.cc:64","thread":"29"}
{"time":"2026-06-05T11:32:08.443373Z","level":"debug","scope":"envoy connection","msg":"closing socket: 0","caller":"external/envoy/source/common/network/connection_impl.cc:313","thread":"29","ConnectionId":"521"}
{"time":"2026-06-05T11:32:08.443379Z","level":"debug","scope":"envoy connection","msg":"onBelowWriteBufferLowWatermark","caller":"external/envoy/source/common/network/connection_impl.cc:631","thread":"29","ConnectionId":"521"}
{"time":"2026-06-05T11:32:08.443384Z","level":"debug","scope":"envoy http","msg":"Read-enabling downstream stream due to filter callbacks.","caller":"external/envoy/source/common/http/conn_manager_impl.cc:2065","thread":"29","ConnectionId":"520","StreamId":"245721652559981291"}
{"time":"2026-06-05T11:32:08.443396Z","level":"debug","scope":"envoy http2","msg":"Stream 1 enabled, unconsumed_bytes 1047100 read_disable_count 1","caller":"external/envoy/source/common/http/http2/codec_impl.cc:535","thread":"29","ConnectionId":"520"}
{"time":"2026-06-05T11:32:08.443425Z","level":"debug","scope":"envoy client","msg":"disconnect. resetting 1 pending requests","caller":"external/envoy/source/common/http/codec_client.cc:115","thread":"29","ConnectionId":"521"}
{"time":"2026-06-05T11:32:08.443430Z","level":"debug","scope":"envoy client","msg":"Request reset. Reason 7","caller":"external/envoy/source/common/http/codec_client.cc:167","thread":"29","ConnectionId":"521"}
{"time":"2026-06-05T11:32:08.443436Z","level":"debug","scope":"envoy connection","msg":"Not closing conn, socket is not open","caller":"external/envoy/source/common/network/connection_impl.cc:148","thread":"29","ConnectionId":"521"}
{"time":"2026-06-05T11:32:08.443443Z","level":"debug","scope":"envoy router","msg":"upstream reset: reset reason: connection termination, transport failure reason: ","caller":"external/envoy/source/common/router/router.cc:1505","thread":"29","ConnectionId":"520","StreamId":"245721652559981291"}
{"time":"2026-06-05T11:32:08.443471Z","level":"debug","scope":"envoy router","msg":"Executing cleanup(): resetting retry_state_ and disabling timers","caller":"external/envoy/source/common/router/router.cc:1133","thread":"29"}
{"time":"2026-06-05T11:32:08.443482Z","level":"debug","scope":"envoy http","msg":"Sending local reply with details upstream_reset_before_response_started{connection_termination}","caller":"external/envoy/source/common/http/filter_manager.cc:1143","thread":"29","ConnectionId":"520","StreamId":"245721652559981291"}
{"time":"2026-06-05T11:32:08.443516Z","level":"debug","scope":"envoy lua","msg":"coroutine finished","caller":"external/envoy/source/extensions/filters/common/lua/lua.cc:67","thread":"29"}
{"time":"2026-06-05T11:32:08.443600Z","level":"debug","scope":"envoy http","msg":"encoding headers via codec (end_stream=false):\n':status', '503'\n'content-length', '95'\n'content-type', 'text/plain'\n'x-atcloud-response-service', 'api-gateway'\n'x-envoy-peer-metadata-id', 'sidecar~10.202.17.14~api-gateway-7c699cbb9d-56k9t.api-gateway~api-gateway.svc.cluster.local'\n'x-envoy-peer-metadata', 'ChoKCkNMVVNURVJfSUQSDBoKS3ViZXJuZXRlcwqFAQoGTEFCRUxTEnsqeQoUCgNhcHASDRoLYXBpLWdhdGV3YXkKMAofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRINGgthcGktZ2F0ZXdheQovCiNzZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1yZXZpc2lvbhIIGgZsYXRlc3QKJgoETkFNRRIeGhxhcGktZ2F0ZXdheS03YzY5OWNiYjlkLTU2azl0ChoKCU5BTUVTUEFDRRINGgthcGktZ2F0ZXdheQpTCgVPV05FUhJKGkhrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvYXBpLWdhdGV3YXkvZGVwbG95bWVudHMvYXBpLWdhdGV3YXkKHgoNV09SS0xPQURfTkFNRRINGgthcGktZ2F0ZXdheQ=='\n'date', 'Fri, 05 Jun 2026 11:32:08 GMT'\n","caller":"external/envoy/source/common/http/conn_manager_impl.cc:2019","thread":"29","ConnectionId":"520","StreamId":"245721652559981291"}
{"time":"2026-06-05T11:32:08.443647Z","level":"debug","scope":"envoy http","msg":"Codec completed encoding stream.","caller":"external/envoy/source/common/http/conn_manager_impl.cc:2135","thread":"29","ConnectionId":"520","StreamId":"245721652559981291"}
{"time":"2026-06-05T11:32:08.443674Z","level":"debug","scope":"envoy http","msg":"doEndStream() resetting stream","caller":"external/envoy/source/common/http/conn_manager_impl.cc:274","thread":"29","ConnectionId":"520","StreamId":"245721652559981291"}
{"time":"2026-06-05T11:32:08.443697Z","level":"debug","scope":"envoy http","msg":"stream reset: reset reason: local reset, response details: -","caller":"external/envoy/source/common/http/conn_manager_impl.cc:2091","thread":"29","ConnectionId":"520","StreamId":"245721652559981291"}

This seems to be worse when i intentionally slow my connection down, adding weight to the theory of it it being about body consume.

Furthermore when we added body consumption in the application before returning the 401, the issue went away.

I am presuming what's happening here is the connection termination is causing the 503UC to be recorded and returned regardless of the response headers (401) that had been sent. This feels to me like it should actually be a 401UC, not a 503UC.

Version

`1.29.3`

Additional Information

No response

Source: istio/istio

View original on GitHub View discussion on GitHub