Flaky test Triage - 2026-09-17

This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them.

This triage covers the 14 commits merged to master on 2026-09-16. Two commits had at least one failed job (two failed jobs in total): one push-triggered unit-test shard and the scheduled Cron Job ITs run whose head was the last commit of the day. Neither failure is caused by the commit itself: one is a known count-snapshot race in TaskQueueScaleTest that already failed on other PRs and passed on the neighbouring master commits before and after (and on the PR's own pre-merge run), and the other is the OWASP dependency-check job flagging memcached server CVEs against the Java memcached client jar, a false-positive CPE match that has failed every cron run since at least 2026-09-14. Both already have fixes in flight (#20291 and #20236). No re-runs had been triggered on either failed job.

Summary

Commit	Failed job	Failure log	Root cause	Verdict
3958b1fac4 (#20347, build(deps): bump com.netflix.spectator:spectator-api)	unit tests (25, H*,D*,T*,O*)	job 104633459736	TaskQueueScaleTest.doMassLaunchAndExit: running+pending+waiting count 974/998/996/999 vs expected 1000, failed all 4 surefire attempts	Flaky (race) (fix in #20291)
5b1bcc1178 (#20316, test: fix flaky KubernetesTaskRunnerTest start tests)	security vulnerabilities (cron)	job 104651620993	dependency-check-maven:13.0.0:check on druid-server: elasticache-java-cluster-client-1.2.4.jar matched to cpe:2.3:a:memcached:memcached:1.2.4 and flagged with 10 memcached server CVEs (CVE-2016-8704 9.8, CVE-2016-8706 8.1, CVE-2023-46853 9.8, CVE-2026-47783 8.1, CVE-2026-47784 8.1, ...)	Persistent (fix in #20236)

Analysis and suggested fixes

1. TaskQueueScaleTest.doMassLaunchAndExit (indexing-service, H*,D*,T*,O* shard)

The test adds 1000 NoopTasks to a TaskQueue whose management loop runs every 1 ms and hands tasks to a TestTaskRunner. Immediately after the add loop it computes getRunningTaskCount(), getPendingTaskCount() and getWaitingTaskCount() as three separate calls and asserts that their sum equals 1000. "Waiting" is defined as "active in the queue but not yet known to the runner", and "pending" as "known to the runner but not yet running". While the management thread is still submitting tasks, a task can move from waiting to pending between the pending snapshot and the waiting snapshot, so it is counted in neither. The test's own comment (in theory we can get a race here, since we fetch the counts at separate times) acknowledges this. The shortfall of 26/2/4/1 tasks across the four attempts matches how far the submission loop advanced between the two snapshots, and the fact that every attempt missed a different number confirms a timing race rather than a logic bug.

The commit (#20347) only bumps spectator-api in extensions-contrib/spectator-histogram/pom.xml and does not touch indexing-service. The same shard passed on the PR's own pre-merge run and on the master commits immediately before (25f56955e8) and after (0f7071077f). The identical failure was triaged in #20312 on a different PR (#20268), so this is a recurring flake.

Suggested fix: merge #20291, which removes the summed three-snapshot assertion right after the add loop (keeping the getTasks().size() == 1000 check, which reads a single ConcurrentHashMap) and replaces the fixed Thread.sleep(100) before the final zero-count assertions with a bounded poll until active/running/pending/waiting are all empty. If a positive-count assertion is still desired at the start, take a single consistent snapshot instead, for example by asserting getTasks().size() == numTasks and separately that each per-datasource counter is <= numTasks, or poll until running + pending + waiting == numTasks within the test's 60 s timeout.

2. security vulnerabilities (cron): false-positive memcached CVEs on elasticache-java-cluster-client-1.2.4.jar

OWASP dependency-check maps com.amazonaws:elasticache-java-cluster-client:1.2.4 to cpe:2.3:a:memcached:memcached:1.2.4, the C memcached server daemon, purely because the version strings coincide. All reported CVEs (including the recently published CVE-2026-47783 and CVE-2026-47784) are server-side memcached vulnerabilities; Druid only uses this jar as a client library in the memcached cache extension and never runs a memcached process. The dependency was bumped from 1.2.0 to 1.2.4 by Dependabot in #20026, and the version-specific CPE match started after that. There is still no suppression entry for this artifact in owasp-dependency-check-suppressions.xml on master. The job has failed on the cron runs of 2026-09-14, 2026-09-15 (see #20360) and now 2026-09-16, so it is red on every recent run and unrelated to #20316 (a KubernetesTaskRunnerTest test-only change), which merely happened to be the head of master when the cron fired.

Suggested fix: merge #20236, which adds a <suppress> entry matching ^pkg:maven/com\.amazonaws/elasticache-java-cluster-client@.*$ against cpe:/a:memcached:memcached with an explanatory note. If #20236 is blocked on its other content, split that single suppression block into a standalone PR so the cron job goes green again.