[Serve] [RFC] Computed Retry-After for backpressure rejections

Motivation

PR #65193 added BackpressureConfig. Deployments can now reject backpressured requests with a 429 and a static Retry-After header. Each caller (proxy or DeploymentHandle) has its own request queue, and at the time of a backpressure rejection a static Retry-After value can be incorrect in both directions. If the queue is deep and the static value is too short, clients retry into another rejection, adding load exactly when the system is trying to shed it. If the queue drains fast and the value is too long, capacity sits idle while clients wait.

This proposal computes the Retry-After value from observed load. This makes the header a reliable estimate of when capacity will be available, instead of a number the operator guessed. This was the original shape of the production request behind #65193 and it fits the discussion there about making 429 the default at some point.

No OSS inference server does this today (TGI, Triton, NVIDIA Dynamo and TorchServe all reject without a retry hint). The closest prior art is cloud-side (Azure OpenAI's retry-after-ms, AWS's x-amz-retry-after) and per-client rate limiters, where the value can be exact because a rate window is a clock. Capacity recovery is not a clock, so here the value is an estimate at any layer.

Goals

• An opt-in BackpressureConfig policy that computes Retry-After from load signals the rejecting component already has.
• Keep the rejection path as cheap as it is today. No RPCs, locks or extra work on reject-and-respond.
• Degrade gracefully: computed, else static, else no header. Never fail or delay a rejection.

Non-goals

• Changing when requests are rejected. Admission control is a separate discussion.
• Guaranteeing admission at the suggested time. The header is a hint, not a reservation.
• Changing the default rejection status code. Tracked separately.
• Implementing any further policy, such as the cluster-informed one (see "Future policies").

Proposed API

from ray.serve.config import BackpressureConfig

@serve.deployment(
    max_queued_requests=64,
    backpressure_config=BackpressureConfig(
        status_code=429,
        retry_after_policy="queue_drain_rate",   # NEW: "static" (default) | "queue_drain_rate"
        retry_after_s=5,                         # static value; also the fallback for computed policies
    ),
)

retry_after_policy is a named policy instead of a bare "auto" so future strategies can be added without changing what existing configs mean.

Proposed design

This design adds a new field to BackpressureConfig called retry_after_policy, and one new policy named queue_drain_rate that computes Retry-After from the rejecting component's queue cap and its observed drain rate.

The overall mechanism has 3 basic aspects: (i) how to compute the Retry-After value (what the queue_drain_rate policy implements), (ii) where to compute it, and (iii) who stamps the header and emits it to the client.

These boil down to emission and estimation. Emission answers (iii) and is fixed: the rejecting component stamps the header. Estimation covers (i) and (ii): one shared implementation, invoked at the two places Serve rejects today, each feeding its own local signals.

The queue_drain_rate policy (this proposal)

retry_after = ceil(max_queued_requests / drain_rate)

• The numerator is the configured cap, not a measurement. A rejection only fires once the queue has reached max_queued_requests, so the depth at rejection is always that value.
• drain_rate is the live signal: how fast the queue empties, updated at the end of every interval as drain_rate = 0.3 * rate_over_last_interval + 0.7 * previous_drain_rate. rate_over_last_interval is the fresh measurement: requests drained during the interval that just ended. previous_drain_rate is this same formula's output from the previous update, so it carries the whole history with older intervals fading out. The 70% weight on the trend keeps lumpy traffic from whipping the value around (completions land in bursts, so a single interval can read 0 or 3x the true rate), and the 30% on the fresh measurement keeps the estimate chasing real load changes within a few intervals. The split, the interval length and the warmup threshold are implementation constants with environment-variable overrides, like Serve's other internal tunables, not new BackpressureConfig fields. The update runs on a lightweight background timer in the component's existing metrics machinery (default 1s). An interval with zero completions counts as a real observation when work is in flight, since that is what slow drain looks like, but idle intervals (nothing running or queued) skip the update instead of decaying the rate. The rate counts as warm after a configurable minimum number of counted intervals; until then the fallback applies.
• Two call sites, one implementation. On the proxy path the router rejects; its drain rate is its own assignment throughput (already counted per replica today), which is fleet-blended: faster replicas take more assignments, so mixed hardware is priced in. On direct ingress the replica rejects; its drain rate is its own completion rate, and its queue is drained only by itself. Same formula, different local inputs, different values, each describing its own queue. A replica's value is a conservative upper bound, since a retry may be re-balanced to a replica that frees up sooner.
• Clamp to [1, 60] seconds. Common client SDKs ignore Retry-After above 60s.
• Jitter (for example +/-20%) is applied in the shared header helper, so the static retry_after_s path gets it too. Without spread, clients refused together retry together. This also keeps the acceptance A/B fair. (Credit: review feedback on this issue.)
• Fallback ladder: computed value (if warm), else static retry_after_s (if set), else no header. A policy failure downgrades silently.
• Format: RFC 9110 delta-seconds (integer, rounded up), same as today.
• Observability: emit the suggested delay and compare it against observed local drain time. Client retries can't be reliably correlated (new request IDs, different proxies, or no retry), so local drain time is the primary measure and admitted-retry rate is secondary where correlation exists.

Future policies (not in scope)

The policy field makes new strategies additive. One example is a cluster_informed policy: the controller already aggregates cluster-wide load for autoscaling and broadcasts per-deployment state over long-poll, so it could supply the signals no rejecting component sees locally (queued demand at other callers, and time-to-new-capacity when a scale-up is in flight). It would slot in as one more input at the top of the same fallback ladder, read only from a local cache so rejections never wait on the controller. It would help the replica vantage most, since a replica sees only itself. Not proposed here: the value would be stale by the metrics cadence, it needs its own anti-herding treatment, and this proposal's observability is the evidence to decide whether it earns that complexity.

Alternatives considered

• Jittered static only (status quo plus jitter): jitter spreads retries around a center, but the center is still a guessed constant, and drain rate moves 10-100x across load regimes, so the constant is wrong exactly when it matters. The acceptance A/B compares against exactly this baseline.
• Client-side adaptive backoff only: SDK backoff can't see queue state. The server is the only party that knows.
• Expose raw signals in headers (queue depth, capacity, per the IETF RateLimit headers draft): pushes the estimation problem onto every client. Could complement a computed Retry-After later, not replace it.

Open question

What makes the computed value "good"? The acceptance test is an A/B of queue_drain_rate against a jittered-static baseline under the same workloads, measured primarily as suggested delay vs observed local drain time, with admitted-retry rate secondary where retries can be correlated. We are looking for input on the target. It could also become a per-deployment knob later, since latency-critical apps prefer optimistic values (fail over fast) and transactional apps prefer conservative ones (retry once, succeed).

cc @edoakes @zcin @abrarsheikh @YashwanthRanjanSingaravel @brent-anyscale @RehanSD