Guidance needed: increasing SQEs per `io_uring_enter` for a request/response client workload

Summary

We migrated an Aerospike client I/O path onto the core netty-transport-native-io_uring transport, hoping to reduce syscalls versus epoll. Functionally it works well. However, the vast majority of our io_uring_enter calls carry exactly 1 SQE, so the syscall-per-operation ratio is close to 1:1 and we have not realised the batching benefit we were hoping for.

We believe we understand why from reading the Netty source (analysis below), but we would appreciate a sanity check on that understanding, and guidance on whether there is any supported way to increase submission batching for this kind of workload — or whether this is simply the expected floor and we should stop looking.

Environment

Event loop group is constructed as:

new MultiThreadIoEventLoopGroup(threads, IoUringIoHandler.newFactory())

and handed to Aerospike's NettyEventLoops, which round-robins commands across the loops.

Workload shape

This is a latency-sensitive ad-serving service. The relevant characteristics:

• ~600 QPS of user-profile lookups per pod (mean inter-arrival ~1.67 ms).
• Each lookup is an Aerospike batch read of up to 6 keys.
• The Aerospike client splits those keys by server node, producing one command (one socket write) per distinct node touched — typically 5-7 commands per request.
• Responses arrive independently from each of those 5-7 nodes, at different times, on different sockets.
• Requests originate from a Jetty thread pool (thread-per-request), so calls into the event loop always come from outside the event loop thread.

What we observe

strace on the event loop thread, steady state:

io_uring_enter(0, 1, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1
io_uring_enter(0, 1, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1
io_uring_enter(0, 6, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 6
io_uring_enter(0, 1, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1
io_uring_enter(0, 1, 0, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1
io_uring_enter(0, 1, 1, IORING_ENTER_GETEVENTS|IORING_ENTER_REGISTERED_RING, NULL, 8) = 1

Roughly 3150 enters/sec on one loop. The distribution is dominated by = 1, with occasional = 5/= 6/= 7 groups. Depth never approaches the 128-entry ring.

Our reading of this, per request:

600 QPS × ~7 ≈ 4200 enters/sec, which is the right order of magnitude for what we measure.

Our understanding of the mechanism

From reading 4.2.11 sources — please correct us if any of this is wrong:

1. The write side already batches optimally. The Aerospike client dispatches all per-node commands for one request back-to-back from the calling thread. Only the first execute() writes the eventfd, because IoUringIoHandler#wakeup coalesces via eventfdAsyncNotify.getAndSet(true). By the time the loop wakes, all 5-7 tasks are queued, runAllTasks drains them together, and we get a single multi-SQE enter. This is the = 6 line above and it is exactly the behaviour we want.

2. The read side cannot batch, structurally. Each response is a separate CQE arriving at a separate time from a separate socket. Handling it enqueues a re-arm SQE, and then in processCompletionsAndHandleOverflow the next pass sees p == 0 with needSubmit() true and calls submitAndClearNow0 → submitAndGetNow() → submitAndGet0(0) → submit(1, 0, IORING_ENTER_GETEVENTS). That is one enter carrying one SQE, per response. These are ~85% of our enters.

3. Ring size is irrelevant here. SubmissionQueue#enqueueSqe only force-submits at pending == ringEntries, but IoUringIoHandler#run submits at the end of every loop iteration regardless of depth. So the batch size per enter is "whatever accumulated during one iteration", which at our duty cycle is ~1. To fill 128 entries we estimate we'd need something north of 400k QPS on a single loop.

In other words: there appears to be no "wait until N SQEs are pending before entering" mode, and with independent response arrivals there is nothing to accumulate anyway.

What we have already tried (and what happened)

Questions

1. Is our analysis correct, particularly point 2 — that per-response single-SQE enters are inherent to a request/response client workload with independent completion arrivals, and not something we've misconfigured?

2. Is there any supported way to defer or coalesce submissions? Something equivalent to "don't call io_uring_enter until either N SQEs are pending or T microseconds have elapsed". We could not find one. Is such a knob something Netty would consider, or is it deliberately avoided because of the latency cost?

3. IORING_SETUP_SQPOLL — we note it is not implemented (no IORING_SETUP_SQPOLL constant in Native, no IORING_ENTER_SQ_WAKEUP, though IORING_SQ_NEED_WAKEUP is defined at Native.java:257 but never read). Is that a deliberate design decision? Our own assessment is that it would be a poor fit for us — a kernel polling thread spinning per ring against a loop that is idle ~97% of the time would trade syscalls for a dedicated core, which is the wrong direction since our actual goal is reducing CPU. But we'd value your view on whether SQPOLL is on the roadmap and what workload shape you'd consider it appropriate for.

4. IORING_SETUP_DEFER_TASKRUN — we understand this is enabled automatically via Native#setupFlags when singleIssuer is true (the default) and the kernel supports it. We are adding startup logging of IoUring.featureString() to confirm it is actually active on our kernel. Are there other setup flags or system properties you'd recommend for a many-small-request client workload like this?

5. Is ~1 syscall per network operation simply the expected floor here, with io_uring's benefit for our shape being reduced per-syscall cost and better completion handling rather than fewer syscalls? If so we're happy to accept that — we'd just like to stop chasing it. A clear "yes, that's expected" is a genuinely useful answer.

Non-goals / things we know about

• We are aware we could add an application-level coalescing layer — buffering independent requests for a small time window and merging their keys into one Aerospike batch, so that N requests hitting the same node collapse into one command. This would genuinely help (the client merges keys per node, so command count is bounded by cluster size rather than key count). We're keeping it as a fallback since it trades latency and adds real complexity. This issue is specifically about whether anything at the Netty/io_uring layer can help first.

Thanks very much for the transport work — happy to provide more traces, run experiments, or test patches if that would be useful.