fix(downloads): a single stall fails the job permanently, bypassing attempts: 10
Problem
A single stall permanently fails a download, bypassing the retry policy entirely.
RunDownloadJob is registered with:
attempts: 10,
backoff: { type: 'exponential', delay: 30000 },but there is no maxStalledCount or lockDuration override anywhere in the codebase (grep -rn "maxStalledCount\|stalledInterval\|lockDuration" admin/app admin/config returns nothing relevant). BullMQ's default maxStalledCount: 1 therefore applies, and a stalled job is failed outright rather than consuming an attempt. The ten attempts never come into play.
Observed on v1.34.0-rc.4:
[downloads] Job failed: d395a22f5112af46, Error: job stalled more than allowable limit
[ZimService] Wikipedia download failed for: wikipedia_en_all_mini_2026-06.zimThe worker dies without renewing its lock -> BullMQ marks the job stalled -> maxStalledCount: 1 fails it permanently. The user sees a failed download with a message that explains nothing actionable.
Why this matters beyond a manual restart
The obvious trigger is a container restart mid-download, which sounds like an operator-only concern. It is not:
The sidecar updater recreates the admin container. An auto-update that lands while a user is part-way through a multi-hour download kills the worker and permanently fails the job. Auto-update shipped in 1.33, so this combination is live today. There is precedent for caring about exactly this class of interaction - PR #146 stopped Kiwix being restarted while downloads were running.
Wikipedia Compact is 12.5 GB and the full build is 124 GB, so the window where a download is in flight is measured in hours, not seconds.
Related
Once #1201 lands, the partial .tmp survives and a re-triggered download resumes rather than restarting - which softens the damage considerably. But the job still has to be re-triggered by hand, because it is in a terminal failed state rather than retrying on its own.
Also worth noting when testing: recovery from a stall is not instant. Reclaiming a stalled job took roughly four minutes in observation, so a download can look frozen before it picks up.
Possible directions
Not obvious which is right, hence an issue rather than a PR:
- Raise
maxStalledCounton the downloads queue so a stall consumes an attempt instead of ending the job. Simplest, and makesattempts: 10mean what it appears to mean. - Raise
lockDurationso a briefly-blocked worker is not declared stalled in the first place. Treats a different cause. - Have the updater defer while downloads are active, mirroring PR #146's approach for Kiwix. Addresses the auto-update path specifically but not network-caused stalls.
(1) plus (3) looks like the combination that actually closes the user-visible hole, but the retry semantics deserve a considered choice rather than a quick flag flip.
Found during v1.34.0-rc.4 QA. Not a 1.34.0 blocker - filing for 1.35 triage.
Source: Crosstalk-Solutions/project-nomad