pick_first (legacy): channel permanently stuck in READY with a stale picker after resolver refresh burst during transient network failure; wait-for-ready RPCs black-hole until deadline

grpc-java-pickfirst-wedge.tar.gz

What version of gRPC-Java are you using?

1.76.2 (grpc-netty, netty 4.1.136.Final)

What is your environment?

Linux, Java 17 (also observed under Java 11 in production). Load balancing policy: pick_first (single-address target). Active LB implementation: legacy io.grpc.internal.PickFirstLoadBalancer (its RequestConnectionPicker appears in the channel logs; the new PickFirstLeafLoadBalancer was not enabled — see "Additional data point" below).

What did you expect to see?

After a transient network failure heals, a pick_first channel either recovers or transitions out of READY so that the LB policy or the application (via notifyWhenStateChanged) can react.

What did you see instead?

The channel is left permanently reporting READY while every RPC — including withWaitForReady() RPCs — buffers forever and fails only by its own deadline:

DEADLINE_EXCEEDED: ... Name resolution delay 0 ... [buffered_nanos=..., waiting_for_connection]

No channel state transition ever fires again, so nothing can observe that the channel is broken. ManagedChannel.enterIdle() reliably recovers it.

Mechanism, from FINEST ChannelLogger logs (attached):

1. The transport to the current address dies (TLS handshakes reset — in production, a rolling container upgrade left the port published while the backend reset handshakes for ~10 seconds).
2. gRPC's reconnect logic calls NameResolver.refresh() repeatedly. The resolver answers asynchronously with a mix of onError and onResult deliveries (it is backed by a discovery service that sits behind the same failing network, so deliveries arrive out of order relative to subchannel state changes).
3. Each onResult causes PickFirstLoadBalancer to create a replacement subchannel and schedule delayed shutdown (~5s) of the previous one.
4. Within the delayed-shutdown window, the old subchannel completes its connection against the recovering backend and reaches READY; the LB publishes a READY picker referencing it.
5. The delayed shutdown then fires (SHUTDOWN with UNAVAILABLE(Subchannel shutdown invoked)), terminating the transport — but the published READY picker is never replaced. The channel stays READY forever with a picker whose subchannel has no transport.

Steps to reproduce the bug

Self-contained repro attached: one Java file (BurstRepro.java, only grpc-java + netty dependencies, in-process backend server), a small Python TCP proxy that models the network failure (reset mode: accept + immediately close, emulating TLS resets), and a driver script. Typically reproduces within a few iterations.

Per iteration: verify a healthy RPC through the proxy, switch the proxy to reset for a few seconds (driving reconnect/backoff and resolver refreshes), heal the proxy, then probe with a withWaitForReady() RPC with a deadline.

Two wedge variants observed:

• logs/run2-READY-signature/ — channel READY, wait-for-ready probe black-holes to DEADLINE_EXCEEDED; enterIdle() immediately recovers it. This matches our production incident signature.
• logs/run1-connecting-wedge/ — channel wedged in CONNECTING with no reconnect attempt scheduled.

Additional data point: the new PickFirstLeafLoadBalancer does not wedge

Running the identical repro with GRPC_EXPERIMENTAL_ENABLE_NEW_PICK_FIRST=true (confirmed active: PickFirstLeafLoadBalancer in the logs), the permanent black-hole does not reproduce in 40 iterations. The same post-burst sequence still leaves the channel claiming READY while the picked transport is dead, but the RPC fails fast with UNAVAILABLE and gRPC immediately schedules a resolver refresh, after which the channel recovers (logs/run3-newpf-no-wedge/).

So the catastrophic silent wedge appears specific to the legacy PickFirstLoadBalancer. Since the new PF also has a known stuck-in-IDLE issue (#12395) with a draft fix that appears stalled (#12403), may we suggest revisiting that PR — fixing #12395 and graduating the new PF would retire both wedge variants, whereas today users are stuck choosing between a legacy PF with this READY wedge and an experimental PF with a known IDLE wedge.

Related issues

• #12395 — PickFirstLeafLoadBalancer stuck in IDLE on backend address change (Java analog of grpc/grpc-go#8615). Same family: LB internal state diverging from the published picker when a resolver update races a subchannel state change. Differs in that it wedges in IDLE and affects the new leaf PF; this report is the legacy PF wedging in READY, which is strictly worse (wait-for-ready RPCs black-hole and no observable state ever changes).
• #11082 — PF not emitting TRANSIENT_FAILURE (missed state propagation).
• #11227 — NPE from inconsistent PF internal state after resolution errors (fixed in 1.66); shows the resolver-error path mutating LB state non-atomically.

Production impact

Our etcd-backed auth-service channel black-holed all RPCs for 60+ seconds (until process restart) during a rolling upgrade. The server saw zero incoming RPCs during the window while the client channel claimed READY throughout. We now run an application-level watchdog that calls enterIdle() when RPCs time out while the channel claims READY.

Attachment

grpc-java-pickfirst-wedge.tar.gz — repro sources, driver, and FINEST channel logs for all three runs described above.