Scan All takes over the jobservice worker pool and everything else needs to queue

A Scan All fills the whole jobservice worker pool for hours and every other job type waits behind it, including scans a user just triggered by hand or by push. The two levers gocraft/work already has, priority and a concurrency cap, cannot help because every scan is enqueued under the same IMAGE_SCAN name. Giving scan-all children their own capped, low-priority job name fixes both problems in about 50 lines.

Problem

A Scan All enqueues one IMAGE_SCAN job per artifact with no cap. All job types share the single jobservice worker pool (worker_pool.workers, default 10). Scanning is slow compared to everything else, so a scan-all over a few thousand artifacts holds every worker for hours. Replication, GC, retention, webhooks and any scan a user just triggered by hand or by pushing an image all wait behind it.

Two things go wrong at once. Nothing stops scan-all from taking all N workers. And an ad-hoc scan (manual, multi-select in the UI, scan-on-push) lands in the same Redis list as the scan-all backlog and waits FIFO behind thousands of scheduled jobs. It cannot be prioritised, and pausing the IMAGE_SCAN queue to relieve pressure pauses the ad-hoc scans with it.

Why the existing knobs do not help

gocraft/work has two levers, and Harbor wires both in src/jobservice/worker/cworker/c_worker.go (registerJob):

• Priority (src/jobservice/job/priority.go): every job type is 1000 except DEMO and SLACK at 1. Priority only changes the order in which a worker inspects the queues. It reserves no capacity. If the scan queue is the only non-empty one, every worker takes scan jobs no matter the weights.
• MaxConcurrency (MaxCurrency() on the job interface): src/pkg/scan/job.go:99 returns 0, which gocraft treats as unlimited. A non-zero value is a cluster-wide Redis counter, and at the cap the fetch script moves on to the next queue. That is the behaviour the first problem needs.

Both levers key on the job name. launchScanJob in src/controller/scan/base_controller.go:975 sets Name: job.ImageScanJobVendorType for every scan regardless of trigger. Scan-all children are already distinguishable in Postgres (task vendor_type = SCAN_ALL, inherited from the execution) but not in Redis. So a cap on IMAGE_SCAN blocks ad-hoc scans just as hard, and a priority change cannot tell the two apart.

startScanAll (base_controller.go:459 onwards) runs as a goroutine in core and submits every artifact as fast as the iterator returns them. batchSize := 50 is only the DB page size.

Benefits of fixing it

• A user-triggered or push-triggered scan starts right away, even while a scan-all with thousands of pending jobs is running.
• Replication, GC, retention and webhooks keep their share of the pool during a scan-all instead of stalling for hours.
• Operators get a scan-all-only pause and stop, since the backlog sits in its own queue. Pausing it no longer pauses ad-hoc scans.
• A scan-all becomes safe to schedule on shared or multi-tenant installs, where today it is the single biggest source of jobservice and scanner pile-ups.
• No schema change, no API change, no change to executions, tasks, reports or the UI. The cap defaults to off, so existing installs behave as before until they opt in.

Prior reports

• #23555 Prioritize scan-on-push and manual scans ahead of Scan All (open)
• #23603 Configurable job type queue priority for jobservice (open). The discussion there already concluded that new job names per trigger are needed because gocraft keys on the name.
• #19783 Support configure number of workers for scheduled scan all (open since 2024)
• #19685 raised the Slack queue priority because scans starved it. Closed stale, with a maintainer comment that all queue priorities should be configurable.
• goharbor/community proposals/new/jobservice_monitoring.md (2022) says scan all and similar system tasks "should allocate dedicate workers to run" and left that design out of scope. Nobody picked it up.

None of these has an implementation attached. The options are in the decision record below.

Impact seen in production

On 2026-09-10 a manually triggered scan-all over about 4000 artifacts on a multi-replica install saturated the pool for the whole night. The same run triggered a core DB pool exhaustion. Being able to cap and pause scan-all on its own, without touching user-triggered scans, would have contained both.