PG backend: the stalled checker's unconditional 30s poll keeps an idle database awake

Summary

Split out of #4601 at the request of @0jaspahwa, who is fixing the block-timeout half in #4623.

#4601 was about maximumBlockTimeout capping the blocking wait at 10s. That is real, and #4623 addresses it. But it is not the only unconditional poll on an idle worker: the stalled checker issues a query every stalledInterval (default 30000) for as long as the worker runs, regardless of whether anything is active. On a serverless PostgreSQL that suspends when idle, that timer alone is enough to keep the compute awake forever, so the block-timeout fix does not deliver the idle saving on its own.

Same shape as #4601: a loop whose cost is negligible on Redis, on a shared path, applied to a backend where a round-trip has a price.

The code

All line references are origin/master at 3508972.

The loop is unconditional and untied to whether the worker has any active jobs:

// src/classes/worker.ts:1369
private async stalledChecker() {
  while (!(this.closing || this.paused)) {
    await this.checkConnectionError(() => this.moveStalledJobsToWait());

    await new Promise<void>(resolve => {
      const timeout = setTimeout(resolve, this.opts.stalledInterval);
      ...
    });
  }
}

• Started from run() (worker.ts:580 -> startStalledCheckTimer, :1342), so every ordinary worker has one.
• stalledInterval defaults to 30000 (worker.ts:248) and must be > 0 (:277-279) - there is no "off" value.
• The only exits are closing, paused, or skipStalledCheck (worker.ts:1343).

The PostgreSQL implementation is a round-trip every time, with no client-side short-circuit:

// src/postgres/postgres-queue-backend.ts:1047
async moveStalledJobsToWait(): Promise<string[]> {
  const opts = this.opts as WorkerOptions;
  const { rows } = await this.run<{ id: string }>('move_stalled_jobs_to_wait', [
    this.queueName, opts.maxStalledCount ?? 1, Date.now(), opts.stalledInterval ?? 30000,
  ]);
  return rows.map(r => r.id);
}

The server-side throttle does not help here

move_stalled_jobs_to_wait already throttles itself (src/postgres/migrations/0002_functions.sql:2274), mirroring the Redis stalled-check key with PX maxCheckTime:

SELECT value::bigint INTO v_last
  FROM meta WHERE queue = p_queue AND field = 'stalled-check';
IF v_last IS NOT NULL AND p_now < v_last + p_max_check_time THEN
  RETURN;
END IF;

That is a work throttle, not a wakeup throttle. The client has already opened the round-trip by the time the function decides to return early, so the compute still wakes. With M workers on a queue you get M polls per interval and at most one of them does anything.

Why it matters in practice

Per worker, per queue, on defaults: 2 round-trips a minute, 2880 a day, none of which need to exist on a queue with nothing active.

The consequence is not the query cost, it is the idle window. A provider that suspends after 300s of inactivity (Neon's default, and the deployment in #4601) never sees a 300s gap, so it never suspends. That holds whether the block cap is 10s or 3600s, which is why #4623 landing does not by itself produce the saving #4601 was asking for.

On Redis the identical loop is close to free - the connection is already open and the script is cheap - so this has never needed to be conditional before.

Workarounds today, and what they cost

Both are user-side and both trade something:

• skipStalledCheck: true - turns off stalled recovery entirely. Fine for a queue where no job ever needs reclaiming, not fine in general.
• A large stalledInterval - but the same value is passed to the SQL as p_max_check_time (postgres-queue-backend.ts:1058), so it is both the poll period and the reclaim throttle window. Raising it to let the database sleep makes stalled detection correspondingly slower. It is not a free knob.

What I am not proposing

I do not think "skip the pass when this worker has nothing active" is obviously correct, and I would rather not guess: the whole point of the sweep is reclaiming jobs whose owning worker died, so a locally-idle worker is exactly the one that may still need to run it. Deciding what an idle worker owes the rest of the fleet is a maintainer call, not mine.

Some directions that seem worth weighing, in case they are useful:

• Let the backend own the interval the way #4623 lets it own the block cap, so PostgreSQL can pick something larger than Redis needs.
• Decouple the poll period from p_max_check_time, so the reclaim window can stay at 30s while the poll backs off.
• Fold the sweep into the existing LISTEN-based wake path, so an idle worker is woken to sweep rather than polling for the chance to.

Happy to test any of these against a real Neon instance if a branch appears.

Environment

• bullmq 6.2.0; code above read at origin/master 3508972
• PostgreSQL backend on Neon, suspend_timeout = 300s
• Source-read, not instrumented: the line references and the SQL throttle are checked; the "never suspends" consequence follows from the 30s interval against a 300s idle window rather than from a fresh measurement of my own deployment.

Related: #4601 (block timeout), #4623 (fix for it).