Severe CPS throughput collapse: ReferenceManager$CallBackedManager.removeStallEntries degrades to O(n) under heavy class churn

Severe CPS throughput collapse: ReferenceManager$CallBackedManager.removeStallEntries degrades to O(n) under heavy class churn, stalling all Pipeline threads

Summary

On a high-traffic Jenkins controller (49 workflow jobs, 40-70 pipeline completions/hour, 167 nodes, ~600 concurrent threads), all CPS Pipeline execution periodically collapses to a crawl for several minutes, then self-recovers. A live thread dump captured the mechanism: a CPS thread RUNNABLE inside org.codehaus.groovy.util.ReferenceManager$CallBackedManager.removeStallEntries → ConcurrentLinkedQueue.remove, while every other Pipeline's steps crawl (7s/step vs. the normal ~0ms). Jenkins 2.516.3 / 2.528.3 LTS, Java 17.0.12, Windows Server 2019, -Dgroovy.use.classvalue=true enabled.

We believe this is a structural bottleneck in the bundled Groovy 2.4.21: CallBackedManager.afterReferenceCreation() synchronously calls removeStallEntries0(), which drains the entire ReferenceQueue one reference at a time (poll() → finalizeReference() → clear(), loop). When many classes are being collected (pipeline teardown bursts), the stall-entry list grows and each ManagedReference creation pays an O(n) cleanup cost — and all CPS threads serialize behind it.

Environment

• Jenkins: 2.516.3 (prod) and 2.528.3 (three other controllers) — both bundle groovy-all 2.4.21 (verified: extracted the jar from both controllers, all 4457 classes have identical MD5s, so this is not fixed in current LTS)
• Java: OpenJDK 17.0.12, JVM flags: -Xms32g -Xmx96g -XX:+UseZGC -Dgroovy.use.classvalue=true
• Load profile: ~64.7万 resident classes (other controllers with 10x less traffic: ~25k), Metaspace 2.7 GB, cumulative loaded classes growing at ~160k/day

Evidence

Incident 1 (2026-09-02 13:28) — full outage, 397h uptime

Every new build hung at Obtained ... from git (CPS parseScript). 19 likelyStuck items; zero builds completed after 13:28.

Thread dump: one request thread (a Scriptler script execution, Handling GET /node-label-query) RUNNABLE in:

java.util.concurrent.ConcurrentLinkedQueue.remove
org.codehaus.groovy.util.ManagedConcurrentLinkedQueue$Element.finalizeReference
org.codehaus.groovy.util.ReferenceManager$CallBackedManager.removeStallEntries0
org.codehaus.groovy.util.ReferenceManager$CallBackedManager.removeStallEntries
...

while 88 threads BLOCKED in LockableObject.lock waiting for ClassInfo.getMetaClass. Required a controller restart to recover. /api/json stayed at 12ms throughout (doesn't go through Groovy), so uptime monitors never fired.

Incident 2 (2026-09-03 13:54–14:02) — degraded, only 22h uptime

The external scriptler polling from incident 1 had already been migrated to /computer/api/json (0.04s per call — verified not the trigger). Nevertheless:

• 19 builds across 9 unrelated jobs showed finalization gaps of 1.6–165s (normal: <0.5s; e.g. #77116 gap 165.1s, #77117 gap 161.9s, #24593 gap 149.0s)
• A WorkflowScript thread captured live, RUNNABLE in removeStallEntries → ClassInfo$GlobalClassSet.add → ManagedConcurrentLinkedQueue.add (i.e. even creating a reference pays the cleanup cost)
• A pipeline that was mid-execution during the window crawled at ~7s/step for its final 49 CPS steps; the identical job re-run after recovery executed the same steps at ~0.0s/step
• Two waves ~5 minutes apart, then full self-recovery in ~8 minutes (consistent with the stall queue finally draining)

Decompiled bytecode of the hot path (groovy-all 2.4.21)

// Called on EVERY ManagedReference creation:
public void afterReferenceCreation(Reference r) {
    removeStallEntries();          // synchronous!
}

public void removeStallEntries() {
    ReferenceQueue q = getReferenceQueue();
    if (queuesInProcess.putIfAbsent(q, this) == null) {
        try { removeStallEntries0(q); }          // full drain
        finally { queuesInProcess.remove(q); }
    }
}

private static void removeStallEntries0(ReferenceQueue q) {
    Reference r;
    while ((r = q.poll()) != null) {             // O(n) over ALL stalled refs
        if (r instanceof Reference) {
            Finalizable h = ((Reference) r).getHandler();
            if (h != null) h.finalizeReference(); // runs arbitrary cleanup
        }
        r.clear();
    }
}

Under a burst of class unloading (e.g. after many pipelines finish), one thread drains the whole queue while holding the queuesInProcess slot; every other thread creating a ManagedReference (i.e. every CPS step touching any new MetaClass/CallSite) hits this path. groovy.use.classvalue=true only changes the ClassInfo cache lookup; it does not avoid ReferenceManager cleanup.

Impact

• All pipelines on the controller stall simultaneously, seconds-to-minutes per step, up to full outage
• Self-healing makes it invisible to post-hoc log forensics (Jenkins ring buffer covers ~25 min; GC logs show nothing — we confirmed 0 allocation stalls, max ZGC pause 0.147ms during the incident window)
• Only manifests on high-churn controllers, so it is hard to reproduce on small instances — but any Jenkins at similar scale is exposed

Suggested directions

1. Bound the cleanup: drain at most N references per call in removeStallEntries0 (amortize the rest), so no single afterReferenceCreation pays O(n).
2. Move cleanup off the caller's critical path (dedicated reference-processing thread, like ThreadedReferenceManager which already exists but CallBackedManager is the default for ClassInfo).
3. If a Groovy 2.4.21 patch release is out of scope for Jenkins core, consider bundling a patched build (there is precedent: patched Groovy 2.4.6 in 2016) or backporting GROOVY improvements to the vendored 2.4.21.

Workaround

We are currently planning scheduled low-traffic restarts of the controller to keep the stall-entry accumulation below the pain threshold. Any better known workaround would be appreciated.