No max age / idle timeout for pooled proxy-to-proxy HTTP/2 connections — held connections pin buffer memory until pod restart or OOM

What problem are you trying to solve?

Pooled proxy-to-proxy HTTP/2 connections currently live forever. TCP and HTTP/2 keepalives keep them healthy indefinitely, MAX_IDLE_CONNS_PER_ENDPOINT (default 10,000) only caps the count, the HTTP/1 connection-pool idle timeout doesn't apply to them, and the outbound discovery idle timeout only tears state down when a service receives no traffic at all. We audited the proxy's env config surface from v2.311.0 through v2.366.0 and found no setting that closes a pooled connection for being idle or bounds its lifetime.

Application-level timeouts don't help: our application closes its own idle connections after 5 minutes, but those only govern the app→local-proxy hop. The proxy→proxy pooled connections are owned by Linkerd and persist regardless — so the memory they pin keeps climbing until the pod is restarted manually or gets OOM-killed.

Why this matters for memory: each held server-side connection retains its receive-buffer high-water mark — bounded by the connection window (1MB default) — for its entire lifetime. On a very high-throughput service in our mesh (arm64 EKS), a large caller fleet holds ~300 pooled connections into each server pod. After ~300 rps load bursts we measured ~700KB retained per connection, i.e. ~200MB of proxy RSS per pod, verified via tcp_open_total/tcp_close_total accounting on the admin endpoint. That memory was only ever released when peer pods restarted — never by idling, no matter how long.

How should the problem be solved?

A configurable max connection age (and/or idle timeout) for inbound HTTP/2 server connections: after the configured age, the proxy performs the same graceful shutdown (GOAWAY) it already uses for drain — no new streams accepted, in-flight streams complete, clients reconnect transparently — with per-connection jitter to avoid synchronized shutdown storms. Similar in spirit to gRPC's MAX_CONNECTION_AGE.

We implemented exactly this and have a PR open: linkerd/linkerd2-proxy#4618 (LINKERD2_PROXY_INBOUND_SERVER_HTTP2_MAX_CONNECTION_AGE; unset preserves current behavior; includes an integration test).

Results running it in a production-like mesh with a 15-minute age:

• Proxy RSS became a bounded ~15-minute sawtooth instead of a monotonic plateau — buffers released every cycle with no pod restarts (observed on a build with jemalloc enabled on aarch64 — see #15613 / linkerd2-proxy#4613 — so freed memory was actually returned to the OS).
• Zero failed requests across many recycle cycles; p50/p99 unchanged through recycle waves.
• Cost: one TLS handshake per connection per age period, spread over ~90s by the jitter — below measurement noise in-cluster.

Any alternatives you've considered?

• Scheduled pod restarts — operational toil that masks the issue.
• Lowering the HTTP/2 connection window — caps the per-connection ceiling but not the lifetime; buffers still pin at high-water forever.
• Adaptive flow control (..._ADAPTIVE_FLOW_CONTROL=true) — measured strictly worse in our environment (~3× the memory plateau at lower load; the BDP estimator grows windows past the fixed default under load).
• Application-level connection TTLs — cannot reach the proxy-owned pooled connections, as described above.

How would users interact with this feature?

A proxy environment variable (deliverable via proxy.additionalEnv or the config.linkerd.io/proxy-additional-env annotation), defaulting to unset/disabled so existing behavior is unchanged. Optionally a first-class Helm value/annotation later.

Would you like to work on this feature?

Yes — implementation with tests is already open at linkerd/linkerd2-proxy#4618; happy to iterate on design (naming, defaults, idle-timeout variant, outbound-side equivalent) per maintainer feedback, and to provide graphs and metric snapshots.