Workers stop responding to requests due to Unable to execute HTTP request: Timeout waiting for connection from pool

Author: EvgenyLyCreated May 19, 2026Updated May 19, 2026

Labelstype-bug

Alluxio Version: 2.9.6

Describe the bug We have a cluster of 24 worker nodes. After approximately 6 minutes of operation under a high read load, all threads on the worker are blocked and all new requests fail with errors.

2026-05-19 14:25:12,030 WARN [BlockDataReaderExecutor-761](LogUtils.java:135) - Exception occurred while reading data for read request BlockReadRequest{chunkSize=1048576, end=67108864, id=405874409642, openUfsBlockOptions=ufs_path: "s3://path/description/description__segment0027_0.orc"
offset_in_file: 11408506880
block_size: 67108864
maxUfsReadConcurrency: 32
mountId: 4620979423686187726
no_cache: false
, promote=false, sessionId=344728738333690027, start=0, positionShort=false}. Session 344728738333690027: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool

This behavior only occurs with large ORC files (over 20 Gb).

When the problem occurs, netstat shows about 400 open connections to the s3 endpoint with a high Recv-Q (data is not read from the buffers).

The only way to restore the connection is by restarting the worker.

Worker Parameters

alluxio.underfs.s3.admin.threads.max: 30
alluxio.underfs.s3.upload.threads.max: 80
alluxio.underfs.s3.threads.max: 350
alluxio.worker.network.async.cache.manager.threads.max: 64 
alluxio.worker.network.async.cache.manager.queue.max: 64 
alluxio.worker.network.reader.buffer.size: 16MB 
alluxio.worker.network.block.writer.threads.max: 128 
alluxio.worker.network.block.reader.threads.max: 512 
alluxio.worker.network.keepalive.time: 30s 
alluxio.worker.network.keepalive.timeout: 30s

To Reproduce Steps to reproduce the behavior (as minimally and precisely as possible)

Expected behavior A clear and concise description of what you expected to happen.

Urgency Describe the impact and urgency of the bug.

Are you planning to fix it Please indicate if you are already working on a PR.

Additional context

thread dump of java process contain many threads like

"alluxio-client-netty-event-loop-RPC-24" #1361 daemon prio=5 os_prio=0 cpu=0.60ms elapsed=3024.17s tid=0x00007f16e001eeb0 nid=0x6b9 runnable  [0x00007f0ca33f2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPoll.wait([email protected]/Native Method)
        at sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:120)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:124)
        - locked <0x000010001512ce88> (a io.netty.channel.nio.SelectedSelectionKeySet)
        - locked <0x000010001512cb48> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:136)
        at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
        at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:814)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.lang.Thread.run([email protected]/Thread.java:829)

and

"BlockDataWriterExecutor-9" #2424 daemon prio=5 os_prio=0 cpu=1039.89ms elapsed=2375.45s tid=0x00007f118c0138d0 nid=0x1369 waiting on condition  [0x00007f0f988fe000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x000010002f601e80> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill([email protected]/SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer([email protected]/SynchronousQueue.java:361)
        at java.util.concurrent.SynchronousQueue.take([email protected]/SynchronousQueue.java:920)
        at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run([email protected]/Thread.java:829)

Source: Alluxio/alluxio

View original on GitHub View discussion on GitHub