#43451 macOS: grpc completion-queue poller spins at idle when a channel is spliced through a relay holding half-closed sockets (grpcio 1.83.x)

What happened

Python gRPC applications using the synchronous server (grpc.server(ThreadPoolExecutor(...))) burn 32–59% of a CPU core each, continuously, while serving no work at all, on macOS. The CPU is consumed by the server's completion-queue polling thread, spinning inside grpcio's C core poller. The identical code and shape on Linux idles at ~0.000–0.001 cores per process.

Our shape: worker processes each run a sync gRPC server over an AF_UNIX socket, reached by a parent daemon (a gRPC client) through a unix-socket relay. Each worker also holds a pool of ~32 pre-connected, long-lived, completely idle established connections to its own server (a pre-warmed upstream pool). The relay splices byte-for-byte in both directions and half-closes its write side on EOF (socket.shutdown(SHUT_WR)), so one direction of a spliced socket ends up half-closed while the connection stays established.

The spin appears only once a live client channel is spliced through the relay into the server. Before that, the same process is provably at 0%.

Reproduction steps

We have not yet isolated a minimal standalone script (see "What we have NOT confirmed"), so here is our concrete application shape, which reproduces reliably on macOS dev hosts:

Start a Python process that constructs a synchronous gRPC server on a unix socket:

import concurrent.futures as futures
import grpc

server = grpc.server(futures.ThreadPoolExecutor(max_workers=8))
# ... register a normal service ...
server.add_insecure_port("unix:/tmp/worker.sock")
server.start()
server.wait_for_termination()

Before/while serving, create ~32 established client connections to that server and hold them open, idle, for the life of the process (pre-connected pool; they carry no traffic after being established).
Run a relay process that accepts one more connection, verifies a handshake on it, opens/splices it to a live client channel, and splices both directions with blocking recv/sendall, using shutdown(socket.SHUT_WR) to half-close its write side on EOF. The gRPC client (in our case grpc-go, with default keepalive of minutes and no ping storm) dials through the relay and then holds the channel idle.
Observe with top -l 5 -s 1 -pid <server_pid> on macOS: the server process climbs to ~40–60% CPU and stays there indefinitely, doing no RPC work.

Controls run on the same host, same build, same load, each 0.0% CPU (ps -M deltas of 0.000 s user / 0.000 s sys on every thread over 12 s):

same server entry point without the relay/daemon transport attached;
the full production transport (pre-connected pool included) with no peer connected;
one peer handshake-verified and held open but no channel spliced through.

So none of the app's own Python is spinning — the burn is induced only when a live channel is routed through the relay into the server.

Environment

macOS 26.6.2, 10-core (developer hosts)
CPython 3.13 (cygrpc.cpython-313-darwin.so)
grpcio >= 1.83.1 (pinned by our package; exact runtime micro-version not captured)
Default polling engine on macOS — poll, per the polling engines doc
Server: Python sync server over AF_UNIX; client: grpc-go, channel held idle
Not reproducible on Linux (epoll): a faithful reproduction of the same shape (unix socket, pre-connected idle pool, relay with SHUT_WR half-close, grpc-go client holding the channel, pings every 2 s under PermitWithoutStream) measured 0.000–0.001 cores per worker on grpcio 1.83.1 / CPython 3.13

Measurements

Seven idle worker processes on one macOS host, each running the shape above (no connector/RPC work being served; the only scheduled traffic is a once-per-30 s lightweight probe RPC from the daemon):

worker	lifetime avg %CPU (~2 h uptime)	instantaneous %CPU (top)	threads
1	41.4	24.7 / 61.4 / 60.9	21
2	58.6	16.9 / 48.2 / 48.5	21
3	32.3	23.2 / 47.1 / 43.4	21
4	56.1	26.6 / 42.6 / 54.0	21
5	40.5	20.1 / 39.1 / 48.7	21
6	44.1	20.9 / 49.2 / 40.2	21
7	54.1	22.0 / 55.2 / 58.3	22

(Instantaneous columns are back-to-back top samples; the first sample is top's warm-up.)

Per-thread attribution (macOS libproc thread CPU + /usr/bin/sample call graphs, cross-checked against a known-Python-spinner control): the dominant thread is the synchronous server's CQ loop, with the stack

_PyEval_EvalFrameDefault → cygrpc.CompletionQueue.poll → _internal_poll
  → _next → cq_next → pollset_work → poll

72.2% of that thread's 2 248 samples in a 3 s sample land in poll — a single repetitive syscall leaf, not varied Python execution.
Cumulative thread CPU: 661 s user + 256 s system over ~7 060 s of process uptime (~13% of a core for that thread alone, the largest of 21 threads by far), and it was the only thread in RUNNING state at sample time.
Sibling threads show a high-frequency wake/re-block churn (hundreds of seconds of user time while ~95% parked in __psynch_cvwait), consistent with a pollset waking thousands of times per second.

Aggregate: N idle workers burn ≈ N × 0.4–0.5 cores indefinitely; a host running several such workers forfeits roughly half its capacity doing nothing.

Mechanism hypothesis (hypothesis, not confirmed)

This matches grpc/grpc#26045: on macOS/BSD the poll engine delivers hangup events (POLLHUP) differently than Linux, and a half-closed socket's hangup event goes unhandled, so pollset_work sees the fd as permanently ready and the CQ poller busy-loops at idle. That would explain every observation:

macOS uses the poll engine by default; Linux uses epoll and is quiet at 0.0%.
The trigger is the relay splice, which leaves one direction of the spliced socket half-closed (shutdown(SHUT_WR)); the pre-connected idle pool alone (no channel through the relay) is quiet.
poll returning immediately forever matches the single-syscall-leaf profile of the spinning thread.

We are filing as a new issue rather than commenting on #26045 because ours occurs on current grpcio (1.83.x, 2026) with a unix-socket relay shape, and #26045 is an older report.

What we have NOT confirmed

No C-level stack with fd attribution. py-spy requires root on our macOS hosts, and /usr/bin/sample returns C frames only (no fd/wakeup identity). The specific fd that keeps poll returning immediately has not been named; a spindump/lldb/dtruss capture on a macOS host is pending.
No minimal standalone repro yet. The reproduction section is our application shape, which reproduces on macOS; we have not isolated which element (half-closed socket direction, number of idle connections, unix-socket transport, channel held through the relay) is individually necessary.
Exact grpcio runtime micro-version was not captured (pin is >= 1.83.1).

Happy to run any requested experiment on a macOS host, or capture the spinning pollset state with lldb/spindump if given exact commands.