HttpRemoteTaskRunner: Overlord stalling on huge backlog

Description

HttpRemoteTaskRunner.pendingTasksExecutionLoop() holds the single statusLock monitor while iterating pendingTaskIds, and for every pending task it calls findWorkerToRunTask(Task), which rebuilds a full immutable snapshot of all workers:

// findWorkerToRunTask(Task)
return strategy.findWorkerForTask(
    config,
    ImmutableMap.copyOf(getWorkersEligibleToRunTasks()),   // rebuilt on every call
    task
);

getWorkersEligibleToRunTasks() filters and transforms the whole workers map, and each WorkerHolder.toImmutable() reconstructs that worker's announced-task set via ImmutableWorkerInfo.fromWorkerAnnouncements(...).

O(pendingTasks × workers × tasksAnnouncedPerWorker)

…and the entire pass is executed while holding statusLock.

At small backlogs this is invisible. Under a large pending backlog with the cluster at/near capacity, a single loop pass holds statusLock for many seconds to minutes. Because statusLock is also required by run() (new task submission), taskComplete() / status updates, and the worker-sync path, the Overlord effectively freezes.

Restarting the Overlord does not recover it: the active task set is persisted in metadata and reloaded via syncFromStorage on startup, so pendingTaskIds is immediately large again and the loop re-enters the same lock-holding scan.

Observed on Druid 33.0.0 (httpRemote task runner).

Thread-dump signature at the stall:

• One hrtr-pending-tasks-runner-* thread is RUNNABLE, holding statusLock, deep in ImmutableWorkerInfo.fromWorkerAnnouncements → WorkerHolder.toImmutable → getWorkersEligibleToRunTasks → findWorkerToRunTask → pendingTasksExecutionLoop.
• The other pending-task-runner threads are idle in statusLock.wait().
• TaskQueue-Manager is BLOCKED on the same monitor in HttpRemoteTaskRunner.run(), while holding the TaskQueue giant lock.
• Many Jetty qtp-* handler threads are parked on the TaskQueue lock in OverlordResource.taskPost → TaskQueue.add.

Motivation

Use case: any Overlord using the httpRemote task runner that can accumulate a large pending-task backlog while workers are saturated

Why the change is beneficial:

• No behavior change for correctness: within a single synchronized pass the loop reserves at most one task (it breaks right after workersWithUnacknowledgedTask.putIfAbsent), so the eligible-worker set is invariant across the inner loop — recomputing it per task produces identical results.

Proposed fix:

Compute the eligible-worker snapshot once per pass, not once per pending task. Build it just inside synchronized (statusLock) before iterating pendingTaskIds, and pass it into an overload findWorkerToRunTask(Task, ImmutableMap<String, ImmutableWorkerInfo> eligibleWorkers). This drops the dominant fromWorkerAnnouncements rebuild cost from O(pendingTasks × workers × tasksPerWorker) to O(workers × tasksPerWorker) per pass.

Sketch:

synchronized (statusLock) {
  final ImmutableMap<String, ImmutableWorkerInfo> eligibleWorkers =
      ImmutableMap.copyOf(getWorkersEligibleToRunTasks());   // once per pass

  Iterator<String> iter = pendingTaskIds.iterator();
  while (iter.hasNext()) {
    ...
    immutableWorker = findWorkerToRunTask(ti.getTask(), eligibleWorkers);  // no per-task rebuild
    ...
  }
  ...
}

Affected Version

Reproduced on 33.0.0; code path unchanged on master.

Used some AI to structure the issue.