EventEngine: configurable local-queue wakeups to reduce CPU overhead for unary RPCs

Is your feature request related to a problem? Please describe.

Our traffic consists overwhelmingly of unary RPCs. In a controlled benchmark representing this usage, the EventEngine thread pool's unconditional worker notification after a local enqueue causes substantial CPU overhead.

WorkStealingThreadPoolImpl::Run() adds work to the current worker's local queue when possible, but still calls work_signal_.Signal() after either enqueue path:

https://github.com/grpc/grpc/blob/44ad876be99416b3d1b4842f6b88d1ae8188f80e/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc#L247-L257

The comment explains that this improves performance on 32-core streaming benchmarks with small payloads. We understand the motivation for that trade-off. However, for our predominantly unary traffic and smaller server CPU budget, the additional wakeups and movement of work between threads are expensive.

Version scope: the measurements below use a downstream build based on gRPC 1.60.2, including downstream patches, embedded in a C++ coroutine-based server. We verified that the unconditional local-enqueue signal is still present in master at the commit linked above, but we have not benchmarked that master commit. Its polling-cycle implementation has also changed, so the measured improvement must not be treated as a demonstrated result for current master.

Describe the solution you'd like

Please make this wakeup policy configurable, at least as an experimental option scoped to an EventEngine/thread-pool instance, while preserving the existing policy as the default.

A useful alternative policy would prefer execution by the current worker for locally queued work when it can safely make progress, instead of always notifying another worker. Enqueues from outside the pool should continue to notify workers normally.

An adaptive policy that considers local backlog and available parallelism could also address this. The key request is a supported way to choose a lower-overhead policy for unary-heavy applications without maintaining a private thread-pool patch.

This request is about the wakeup policy, independently of the maximum thread count. Any alternative needs to preserve progress when polling tasks block, fairness, shutdown, and the streaming workloads that motivated the existing behavior.

Describe alternatives you've considered

• Keeping the current policy and reducing the thread-pool size: this can reduce part of the overhead, but leaves the unconditional local-enqueue notification in place.
• Using iomgr for the server transport in our current integration.
• A diagnostic-only change that skips the signal for local enqueues. This isolated the effect below; we are not proposing this early return as an unconditionally safe production fix.

Additional context

Controlled A/B experiment

We built one instrumented executable with a runtime switch. In its experimental mode, the switch returns after adding the closure to the local queue only when the caller is a worker of that same pool (g_local_queue != nullptr && g_local_queue->owner() == this). It skips the subsequent notification; the task is still queued and executed through the normal worker loop. Global-queue notifications remain enabled.

The switch does not change thread counts, socket notifications, read/write algorithms, RPC processing, or client behavior. Counters for local/global enqueues and skipped notifications are present in both modes.

Setup:

• Linux x86_64, kernel 5.15.0-143-generic, 12-vCPU VM.
• Server pinned to 4 CPUs; client pinned to the other 8 CPUs. The gRPC pool starts with 12 workers in both EventEngine modes. The embedding server has 4 application workers and 2 event-loop threads.
• C++ callback API, typed unary echo server, 8 Unix-domain-socket connections.
• Generic callback client with a pre-serialized request and ByteBuffer responses, without client-side Protobuf decoding.
• 1000 RPC/s total, paced independently of completion, with 64-byte or 1-MiB request/response payloads. This tests a steady, moderate request rate.
• Two seconds of warmup followed by ten seconds of request submission and completion of outstanding calls.
• Two repetitions with reversed variant order. Four modes: iomgr and EventEngine, each with the diagnostic switch off and on. All 16 runs used the same server executable and completed 160,000 measured RPCs with no RPC errors or missed submissions.

Medians of the two runs, server CPU time measured as process user + system CPU divided by completed RPCs:

Payload	iomgr CPU, µs/RPC	EventEngine CPU, µs/RPC	EventEngine with local notification suppressed, µs/RPC	EventEngine OS context switches/RPC, before → after
64 B	192.96	259.97	178.13	6.22 → 3.32
1 MiB	1179.57	1870.30	1283.58	59.81 → 10.51

Suppressing the local notification reduced EventEngine CPU time by approximately 31% for both payload sizes. On 1 MiB, the paired comparison removed 78–90% of the CPU gap relative to iomgr. The remaining difference has not been conclusively attributed.

As a negative control, applying the same switch to iomgr barely exercised this pool's local-enqueue path and did not produce a comparable improvement (1-MiB CPU: 1179.57 → 1173.88 µs/RPC).

Separate syscall measurements

Separate perf stat runs on the diagnostic EventEngine binary support the wakeup explanation. These counts include warmup and are normalized by all 12,000 RPCs in each profiling window; they are separate from the unprofiled CPU A/B results above.

Payload	Counter per RPC	Existing policy	Local notification suppressed
64 B	recvmsg	3.083	3.083
64 B	sendmsg	1.024	1.024
64 B	epoll_wait	2.181	2.178
64 B	futex	8.186	2.215
1 MiB	futex	65.841	3.745
1 MiB	CPU migrations	16.475	0.575
1 MiB	epoll_wait	25.559	8.420

For small messages, socket syscall counts remain essentially identical while futex traffic drops. For large messages, scheduling also changes how readiness events and I/O work accumulate, despite leaving the read/write implementations unchanged.

The measurements come from our integration harness; a standalone upstream reproducer is not attached. Two repetitions are enough to motivate this request but do not establish a universal performance result. In particular, these results do not cover highly concurrent streaming workloads or demonstrate that simply removing the notification is safe in every polling-cycle implementation.

The broader pool-sizing discussion in #37180 is related, but this request specifically concerns the policy for notifying workers when work was placed in a local queue.