fix(downloads): a single stall fails the job permanently, bypassing attempts: 10

Author: chriscrosstalkCreated Aug 3, 2026Updated Sep 13, 2026
Labelsreleased on @rc

Problem

A single stall permanently fails a download, bypassing the retry policy entirely.

RunDownloadJob is registered with:

typescript
attempts: 10,
backoff: { type: 'exponential', delay: 30000 },

but there is no maxStalledCount or lockDuration override anywhere in the codebase (grep -rn "maxStalledCount\|stalledInterval\|lockDuration" admin/app admin/config returns nothing relevant). BullMQ's default maxStalledCount: 1 therefore applies, and a stalled job is failed outright rather than consuming an attempt. The ten attempts never come into play.

Observed on v1.34.0-rc.4:

[downloads] Job failed: d395a22f5112af46, Error: job stalled more than allowable limit
[ZimService] Wikipedia download failed for: wikipedia_en_all_mini_2026-06.zim

The worker dies without renewing its lock -> BullMQ marks the job stalled -> maxStalledCount: 1 fails it permanently. The user sees a failed download with a message that explains nothing actionable.

Why this matters beyond a manual restart

The obvious trigger is a container restart mid-download, which sounds like an operator-only concern. It is not:

The sidecar updater recreates the admin container. An auto-update that lands while a user is part-way through a multi-hour download kills the worker and permanently fails the job. Auto-update shipped in 1.33, so this combination is live today. There is precedent for caring about exactly this class of interaction - PR #146 stopped Kiwix being restarted while downloads were running.

Wikipedia Compact is 12.5 GB and the full build is 124 GB, so the window where a download is in flight is measured in hours, not seconds.

Related

Once #1201 lands, the partial .tmp survives and a re-triggered download resumes rather than restarting - which softens the damage considerably. But the job still has to be re-triggered by hand, because it is in a terminal failed state rather than retrying on its own.

Also worth noting when testing: recovery from a stall is not instant. Reclaiming a stalled job took roughly four minutes in observation, so a download can look frozen before it picks up.

Possible directions

Not obvious which is right, hence an issue rather than a PR:

  1. Raise maxStalledCount on the downloads queue so a stall consumes an attempt instead of ending the job. Simplest, and makes attempts: 10 mean what it appears to mean.
  2. Raise lockDuration so a briefly-blocked worker is not declared stalled in the first place. Treats a different cause.
  3. Have the updater defer while downloads are active, mirroring PR #146's approach for Kiwix. Addresses the auto-update path specifically but not network-caused stalls.

(1) plus (3) looks like the combination that actually closes the user-visible hole, but the retry semantics deserve a considered choice rather than a quick flag flip.

Found during v1.34.0-rc.4 QA. Not a 1.34.0 blocker - filing for 1.35 triage.

Source: Crosstalk-Solutions/project-nomad