[Bug] MuTransporter.ask creates an unbounded number of scheduler threads
Describe the bug
When the MU protocol is enabled, every call to MuTransporter.ask creates a new single-threaded scheduled executor to handle the request timeout.
The executor reference is discarded immediately and the executor is never shut down. Its core thread therefore remains alive after the request completes. Continuous MU ask traffic causes the number of pool-N-thread-1 threads and native thread memory usage to grow until the application can reach an OOM.
The problematic code is in:
powerjob-remote/powerjob-remote-impl-mu/src/main/java/tech/powerjob/remote/mu/MuTransporter.java
Executors.newSingleThreadScheduledExecutor().schedule(...);To Reproduce
- Start a PowerJob 5.1.2 worker or server with the MU protocol enabled.
- Continuously send requests through
MuTransporter.ask. - Monitor JVM threads with
jcmd <pid> Thread.printorjstack <pid>. - Observe that each request creates a new thread named
pool-N-thread-1. - After the timeout task executes or is no longer needed, the executor thread remains waiting in
ScheduledThreadPoolExecutor$DelayedWorkQueue.
In one production observation, the process had 4,402 live threads, including 3,631 default pool-* threads and 3,583 unique pool IDs. Most of these threads were waiting in empty ScheduledThreadPoolExecutor$DelayedWorkQueue instances. The live thread count later increased to 4,725 without restarting the process.
Expected behavior
MU requests should use a bounded scheduler owned by the MuTransporter instance instead of creating one executor per request. A completed request should cancel its timeout task, and cancelled tasks should be removed from the scheduler queue.
Applications that do not use the MU protocol should not create the MU timeout scheduler or its thread.
Environment
- PowerJob Version: 5.1.2
- Java Version: OpenJDK 11 (the affected module also targets Java 8)
- OS: Linux
Screenshots
Not applicable.
Additional context
The same per-request executor creation is present on the current master branch at commit 332179de (v5.1.2). A single instance-level ScheduledThreadPoolExecutor with setRemoveOnCancelPolicy(true) is sufficient to preserve the existing asynchronous timeout behavior on Java 8 while preventing unbounded thread growth.
Source: PowerJob/PowerJob