[BUG] Direct buffer OOM on many-core nodes: CopyBytesSocketChannel pins 1MB per IO thread

Describe the bug

CopyBytesSocketChannel allocates a 1 MB direct thread-local buffer per IO thread (opensearch.transport.buffer.size, default 1m, CopyBytesSocketChannel.java:82) on each thread's first read/write, never freed while the thread lives. HTTP and transport share one event-loop group sized by transport.netty.worker_count, which defaults to availableProcessors(), while MaxDirectMemorySize defaults to heap / 2.

So on any node where availableProcessors × 1MB > heap/2, direct memory is exhausted once enough distinct IO threads have served traffic:

java.lang.OutOfMemoryError: Cannot reserve 1048576 bytes of direct buffer memory (allocated: 268029480, limit: 268435456) at org.opensearch.transport.CopyBytesSocketChannel.lambda$static$0(CopyBytesSocketChannel.java:82) at org.opensearch.transport.CopyBytesSocketChannel.doReadBytes(CopyBytesSocketChannel.java:139) Observed on a 384-CPU node with 512m heap: ~255 event-loop threads × 1 MB pinned the full 256m direct budget; new threads then OOM on every read (a failed ThreadLocal.initialValue is not cached, so it retries per read), with no heap dump since Netty catches the exception.

Related component

Other

To Reproduce

opensearch.yml: node.processors: 100

jvm.options:

-Xms512m -Xmx512m -XX:MaxDirectMemorySize=64m

Start a single node, then open fresh connections sequentially (each new connection is registered on the next event-loop thread, pinning its 1 MB thread-local on first read):

for i in $(seq 1 100); do curl -s "http://localhost:9200/" > /dev/null; done

The OOM appears once ~MaxDirectMemorySize / 1MB distinct threads have been touched (~65 requests here), and repeats on every subsequent request handled by a new thread.

Default config (no overrides): requires availableProcessors × 1MB > heap/2 — e.g. a ≥256-CPU machine with 512m heap, plus any workload that churns enough fresh TCP connections (keep-alive on a single connection will not trigger it).

Expected behavior

The memory should be allocated property without exceptions.

Additional Details

Plugins k-NN

Screenshots no screenshots

Host/Environment (please complete the following information):

• OS: ubuntu
• Version

Additional context N/A