在一个 ClientConn 上同时进行单元 RPC 导致 unix 套接字死锁 (v1.82.1 中出现回退)
作者: chrislusf创建于 2026年8月18日更新于 2026年9月15日
标签P0
What version of gRPC are you using? v1.82.1 through v1.84.0-dev. v1.82.0 and earlier are unaffected. ### What version of Go are you using (go version)? go1.26.4 ### What operating system (Linux, Windows, …) and version? Reproduced on both linux/arm64 (golang:1.26 container) and darwin/arm64. ### What did you do? One ClientConn over a unix domain socket, N goroutines each looping unary RPCs. All gRPC options are defaults. Repro is a single file with no codegen (raw []byte codec + hand-written ServiceDesc), attached below. go run . -c 64 # ~660k RPCs, fine go run . -c 512 # deadlocks within a second, permanently go run . -tcp -c 512 # ~900k RPCs, fine ### What did you expect to see? RPCs keep completing. ### What did you see instead? Permanent deadlock — no RPC ever completes again. Both directions of the connection stop at once: client http2Client.reader -> controlBuffer.throttle server http2Server.HandleStreams -> controlBuffer.throttle both loopyWriter -> blocked in net.(*conn).Write ### Analysis The change is in internal/transport/controlbuf.go between v1.82.0 and v1.82.1. v1.82.0 had: const maxQueuedTransportResponseFrames = 50 func (*registerStream) isTransportResponseFrame() bool { return false } v1.82.1 replaced this with throttledItem, which registerStream and cleanupStream now embed, so both count toward maxQueuedControlBufferItems (default 100). I instrumented controlBuffer.executeAndPut to dump the queue composition at the instant throttling latches. The budget is consumed almost entirely by those two newly-counted types: latch queued=265 throttled=100 limit=100 : cleanupStream=100 clientHeaders=83 dataFrame=82 latch queued=100 throttled=100 limit=100 : registerStream=98 incomingWindowUpdate=1 ping=1 latch queued=211 throttled=100 limit=100 : cleanupStream=97 clientHeaders=55 dataFrame=56 outgoingWindowUpdate=2 ping=1 registerStream and cleanupStream are per-RPC bookkeeping that produce no outbound wire traffic. Throttling reads on their account means any connection with enough concurrent RPCs latches throttling in both directions simultaneously, and then neither peer can drain the other. Window updates and pings — the frames the throttle was designed for — are 1–2 items. Unix sockets are what make it reachable: their send buffers are small and don't autotune, so loopyWriter blocks in Write readily. On TCP loopback the buffers grow enough that it effectively never happens. GRPC_GO_EXPERIMENTAL_CONTROL_BUFFER_THROTTLE_LIMIT raises the concurrency required but does not remove the failure mode. This is not
内容来源: grpc/grpc-go