Does reset-before-request cover HTTP/1.1 idle connection close/reuse races?

I am testing the race between an upstream HTTP/1.1 server closing an idle keep-alive connection and Envoy reusing that connection for a new request.

In my tests, retry_on: reset recovers these failures, but retry_on: reset-before-request does not trigger any retries.

I would like to clarify whether this is an expected limitation or whether my reproducer is missing a necessary condition.

Related discussions:

• Envoy #10007
• Envoy PR #35074
• Istio #51704

Environment

• Envoy: 1.39.1
• Commit: b579d07d3ad7ee11d32b105e91a5a39ad24718d7
• Linux
• One worker: --concurrency 1
• Request path: curl → Envoy → local Python HTTP/1.1 server
• One static upstream endpoint, plaintext HTTP/1.1
• No additional upstream HTTP filters
• Client retries disabled

Application behavior

The Python server:

1. Returns a complete 200 response with a correct Content-Length.
2. Keeps the connection open without sending Connection: close.
3. Closes the connection after 200ms of inactivity following the response write.
4. Cancels the idle timer when another request arrives and starts a new timer after sending the next response.

Envoy upstream configuration

typed_extension_protocol_options:
  envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
    "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
    common_http_protocol_options:
      idle_timeout: 3s
    explicit_http_config:
      http_protocol_options: {}

Route configuration

route:
  cluster: "inbound|8080||"
  retry_policy:
    retry_on: reset-before-request
    num_retries: 1

Reproduction

I send requests with sleep intervals ranging from 180ms to 200ms, in 1ms increments:

URL='http://127.0.0.1:15006/idle-close?idle_ms=200'

for ((i=1; i<=1000; i++)); do
  curl -q --noproxy '*' --http1.1 --retry 0 \
    --connect-timeout 2 --max-time 5 \
    -sS -o /dev/null \
    -H "X-Request-ID: idle-race-$i" \
    -w "seq=$i code=%{http_code} duration=%{time_total}\n" \
    "$URL"

  sleep_ms=$((180 + (i - 1) % 21))
  printf -v sleep_seconds '0.%03d' "$sleep_ms"
  sleep "$sleep_seconds"
done

The /idle-close endpoint is implemented by the test server as described above.

Each curl invocation creates a new downstream connection. Envoy manages upstream connection reuse independently. Repeated upstream local addresses in access logs indicate that upstream connections are being reused.

The sleep values are approximate; they do not account for curl startup, connection establishment, or scheduling overhead.

Observed behavior

One run with reset-before-request produced:

upstream_rq_total:                        100
upstream_rq_503:                            2
upstream_cx_destroy_remote_with_active_rq:  2
upstream_rq_retry:                          0
upstream_rq_retry_limit_exceeded:           0
upstream_rq_retry_overflow:                 0

Example failure fields:

response_code: 503
response_flags: UC
response_code_details:
  upstream_reset_before_response_started{connection_termination}

request_tx_duration_ms: 0
response_duration_ms: null

With router debug logging enabled, I did not observe performing retry for these failures.

My understanding of the implementation

In UpstreamCodecFilter::decodeHeaders(), the first upstream transmit timestamp is recorded after encodeHeaders() returns successfully:

upstreamTiming().onFirstUpstreamTxByteSent(...);

The router uses that timestamp to determine whether the request has started:

upstream_request_started_ |=
    upstream_request.streamInfo()
        .upstreamInfo()
        ->upstreamTiming()
        .first_upstream_tx_byte_sent_.has_value();

The reset-before-request policy requires:

!upstream_request_started

For an existing HTTP/1.1 connection, with no additional upstream filters, my understanding is:

• If Envoy processes the close event first, it removes the connection from the pool and the next request uses another connection.
• If Envoy processes the request first, acquiring the connection, encoding the headers, and setting the transmit timestamp normally happen without yielding to the event loop. When the close event is subsequently processed, the request is already ineligible for reset-before-request.
• The timestamp does not prove that the application received the request. The encoded data may still be buffered within Envoy.

I also noticed that the positive integration test in PR #35074 uses an upstream filter to pause header encoding before closing the upstream connection.

These are source-based interpretations. I have not yet instrumented upstream_request_started for each failing request.

Questions

1. Is reset-before-request expected to recover HTTP/1.1 idle-close/reuse races when there is no upstream filter pausing header encoding? If so, what specific event sequence or error path makes this possible?

2. Is recording first_upstream_tx_byte_sent_ after successful header encoding an intentional conservative safety boundary? In particular, should retries be rejected even if the bytes have not actually been written to the socket or read by the application?

3. What are the main additional use cases compared with connect-failure? Is an established connection closing while an upstream filter pauses header encoding a primary example?

4. If ordinary idle-close/reuse races are generally outside its coverage, could this limitation be clarified in the documentation? Otherwise, it is easy to interpret reset-before-request as a safe replacement for reset that also addresses these 503 UC failures.

My goal is not to broaden retries at the risk of replaying non-idempotent requests. I would like to understand the policy’s intended coverage and whether it is appropriate as default protection against this particular failure mode.