Flaky test triage - 2026-09-10

Triage of CI failures on the last 20 master commits as of 2026-09-10. 7 commits had at least one failed job (rows for 5e71056845 and 1f68c5e168 added after their runs completed). All failures are flaky tests or CI infrastructure outages, none is caused by the commit itself: every failing test passed on the neighbouring master commits (or on the PR's own pre-merge run of the same shard), and none of the failing runs had been retried.

Summary

Commit	Failed job	Failure log	Root cause	Verdict
5e71056845 (#20268, response identity headers)	unit tests (25, C*)	job 102764139085	CompactionTaskTest.testCompactionWithTimestampDimension (embedded): overlord reported 2 visible segments but SELECT COUNT(*) FROM sys.segments ... on the Broker returned an empty result	Flaky
5e71056845 (#20268)	unit tests (25, T*,F*,G*,J*)	job 102764138933	TaskQueueScaleTest.doMassLaunchAndExit: running+pending+waiting count 983/993/997/999 vs expected 1000, failed all 4 attempts	Flaky (race, fix in #20291)
1f68c5e168 (#19818, harden arithmetic/bounds checks)	unit tests (25, S*)	job 102763954928	SeekableStreamSupervisorStateTest.testSupervisorStopTaskGroupEarly: EasyMock expected TaskQueue.shutdown("id1", ...) once, observed 0 times, failed all 4 attempts	Flaky (race)
8d9545ea58 (#20247, clone sync criteria)	docker-tests	job 102752252468	IngestionDockerTest.test_runIndexTask_andKillData timed out after 240s waiting for segment/schemaCache/dataSource/removed from the broker	Flaky
61ed0a389d (#20272, portable printf)	unit tests (25, R*,B*,Q*,V*)	job 102731457738	ReferenceCountingResourceHolderTest.testResourceHandlerClearedByJVM expected leak count 1, got 3; QueryLaningTest.test_queryUsesLaneInQueryContext_inManualStrategy timed out after 100s waiting for segments to become available	Flaky
61ed0a389d (#20272)	docker-tests	job 102731457503	KubernetesClusterDockerTest setup: k3s pod druid-router-* never became Ready	Flaky / infra
8ff36f7ef8 (#20292, granularity lookups)	security vulnerabilities (cron)	job 102731264787	OWASP dependency-check: NvdApiException: NVD Returned Status Code: 503	Infra
3985d927a6 (#20271, compact object headers)	unit tests (25, K*,E*,W*,Z*,Y*,X*)	job 102727804299	KubernetesTaskRunnerTest.test_start_whenDeserializationExceptionThrown_isIgnored: EasyMock unexpected call peonClient.deleteCompletedPeonJobsOlderThan(...), failed all 4 attempts	Flaky (race)
c7c73625a5 (#20277, jetty bump)	unit tests (25, S*)	job 102713226586	setup_test_profiling_env.sh exited 35 before Maven started: curl SSL connect error downloading jfr-profiler-1.0.0.jar from static.imply.io	Infra

Analysis and suggested fixes

1. KubernetesTaskRunnerTest.test_start_whenDeserializationExceptionThrown_isIgnored KubernetesTaskRunner.start() schedules client.deleteCompletedPeonJobsOlderThan(...) on a real ScheduledExecutorService with a 1 ms initial delay. Whether it fires before the test's verifyAll() depends on thread scheduling, so the strict @Mock peonClient sometimes sees an unexpected call. It failed 4/4 attempts in one run because the retries run in the same loaded JVM. The commit only added a JVM flag to surefire and is unrelated. #20285 hit the same failure independently. Suggested fix: expect the cleanup call with .anyTimes() (or use a nice mock for peonClient); better, inject the cleanup ScheduledExecutorService into KubernetesTaskRunner so tests can supply a manual/no-op executor.

2. ReferenceCountingResourceHolderTest.testResourceHandlerClearedByJVM ReferenceCountingResourceHolder.LEAKED_RESOURCES is a JVM-global static counter and surefire runs with reuseForks=true. The test's System.gc() loop also collects unclosed holders leaked by earlier test classes in the same fork, so the delta is >1. Suggested fix: assert leakedResources() >= initial + 1 together with released.get(), or drop the counter assertion and rely only on the released flag which is specific to this test.

3. QueryLaningTest / IngestionDockerTest.test_runIndexTask_andKillData Both wait on a single emitted metric with a fixed timeout in an embedded cluster on a shared GitHub runner; a slow segment load or a missed emission tick trips it. Suggested fix: poll the actual state (segment availability via sys.segments, or datasource absence in the broker schema) with retries instead of waiting for one metric event, as #19416 already did for the sys.segments part of this test. Raising the 100s timeout in EmbeddedClusterApis.waitForAllSegmentsToBeAvailable is a cheaper stopgap.

4. KubernetesClusterDockerTest (router pod not Ready) Same image build and test passed on the very next master commit; the printf change in the failing commit is behaviour-preserving. Suggested fix: on timeout in K3sClusterResource.waitUntilPodIsReady, include kubectl describe pod output and container logs in the failure message so the next occurrence is diagnosable; consider a longer readiness timeout for the router, which starts last.

5. Security vulnerabilities cron job (NVD 503) Suggested fix: cache the NVD database across runs and/or configure nvdApiKey + nvdMaxRetryCount in dependency-check-maven; treat UpdateException as a soft failure.

6. setup_test_profiling_env.sh exit 35 An external CDN hiccup failed a whole unit test shard before any test ran. Suggested fix: make the profiler download non-fatal, e.g. curl --retry 3 ... || { echo "JFR_PROFILER_ARG_LINE="; exit 0; }.

7. CompactionTaskTest.testCompactionWithTimestampDimension (embedded-tests, C* shard) CompactionTestBase.verifySegmentsCount waits for a single segment/metadataCache/sync/time event on the Broker and then asserts against sys.segments. The first assertion (segment count from the Overlord) passed with 2; only the Broker-side sys.segments query returned nothing, i.e. the sync event that was observed predates the compaction result, or the datasource was not yet in the Broker schema. The commit only adds response headers in Jetty and its pre-merge run of the same C* shard passed. Suggested fix: in verifyNumVisibleSegmentsIs / verifySegmentsCount, retry the sys.segments query until it matches (with a bounded timeout) instead of asserting once after a single sync event.

8. TaskQueueScaleTest.doMassLaunchAndExit (indexing-service, T* shard) The assertion numTasks == running + pending + waiting reads three independent snapshots while tasks are already completing; the test itself carries the comment "in theory we can get a race here". Observed 983, 993, 997, 999 across the 4 attempts, which is consistent with tasks finishing between the snapshots. The same failure was observed on Dependabot PR #20281. #20291 already removes this assertion and waits for cleanup instead of a fixed sleep; merging it should close this one.

9. SeekableStreamSupervisorStateTest.testSupervisorStopTaskGroupEarly (indexing-service, S* shard) The test drains the supervisor's notice queue with while (getNoticesQueueSize() > 0) sleep(100) and then calls runInternal() and verifyAll(). The queue size drops to 0 when a notice is taken, not when it has finished executing, so runInternal() can run before the handoff notice has marked the task group for early stop, and TaskQueue.shutdown(...) is never called. Failed 4/4 attempts under load; the same failure hit PR #20277 on an unrelated jetty bump. The commit only touches test files and does not touch this test or the supervisor. Suggested fix: wait for a completion signal rather than queue emptiness, e.g. expose a "notices processed" counter or latch in the test supervisor, or have handoffTaskGroupsEarly return a future the test can join. Alternatively run the handoff notice synchronously in the test subclass.