Swarm: jobs orchestrator is not stopped on demotion, so replicated jobs run once per ex-leader and then repeat

Description

A Swarm manager that has held raft leadership and then lost it never stops its jobs orchestrator.

In swarmkit's manager/manager.go, becomeLeader() starts the jobs orchestrator alongside the other leader-only components. becomeFollower() stops and nils out the dispatcher, logbroker, CA server, allocator, constraint and volume enforcers, the replicated and global orchestrators, the task reaper, the scheduler, and the role and key managers — but not jobsOrchestrator. It is the only leader-only component that survives demotion. Only Manager.Stop() shuts it down, which in practice means restarting the daemon; cancelling the context does not help, because in that package the context carries logging information only and Stop() is the sole shutdown path.

So in a long-lived cluster, every manager that has ever been leader is still running a jobs orchestrator. They reconcile the same service concurrently, and each creates its own task for the same job iteration. A replicated-job declared MaxConcurrent: 1, TotalCompletions: 1 therefore starts one task per live orchestrator, in the same slot, milliseconds apart — and runs the job's payload that many times.

It then compounds with #42742. ReconcileService computes TotalCompletions - completeTasks - runningTasks in unsigned arithmetic; the overshoot underflows, and the guard returns an error, which abandons the whole reconcile — including the removal of previous-iteration tasks. The job never settles, so the leaked orchestrators create another batch as soon as the first finishes. The job runs over and over until something kills the tasks.

On the affected cluster (five managers, two of them leaking), one deploy produced four tasks, all JobIteration 0, Slot 0:

13:14:00.150  service created
13:14:00.157  task A   node X   complete      <- leaked orchestrator 1
13:14:00.180  task B   node Y   complete      <- leaked orchestrator 2
13:14:00.186  leader: uint64 underflow, we're not going to create 18446744073709551615 tasks
13:17:11.173  task C   node X   (killed)      <- second round, 3ms after the first pair finished
13:17:11.187  task D   node Y   (killed)
13:17:52      leader: uint64 underflow, we're not going to create 18446744073709551613 tasks

The tell on a leaking manager is module=orchestrator/jobs entries on a node that docker node ls does not show as Leader:

level=error msg="error reconciling replicated job" error="raft: failed to process the request: node lost leader status" module=orchestrator/jobs node.id=REDACTED service.id=REDACTED

Those continued every few minutes for hours, including for a service ID that had already been deleted — an orchestrator reconciling a service that no longer exists is the clearest sign it outlived the leadership that started it.

Ordinary replicated services are not affected. A replicated service is a set of slots 1..N, so reconciliation is idempotent and a second orchestrator proposing the same state is a no-op. Only a job's cumulative completion count turns the leak into visible damage. I confirmed this on the affected cluster while two managers were leaking: all non-job services at their desired replica count, one live task per slot, clean rolling-update history.

Workaround: systemctl restart docker on each affected non-leader manager, one at a time. Nothing short of a daemon restart clears a leaked orchestrator. Restarting the current leader is counterproductive — it forces an election and creates a fresh leak on whichever node stands down.

Fix: moby/swarmkit#3295 — two commits: adding the missing m.jobsOrchestrator.Stop() to becomeFollower(), with a test for the general invariant that everything becomeLeader starts is stopped again; and making the reconciler's subtraction saturating so an overshot job stays reconcilable instead of erroring out permanently, which is the #42742 half.

Reproduce

1. Run a swarm with several managers.
2. Cause leadership to move at least once, without restarting the daemon on the manager that stood down (docker node demote / promote, or restart the current leader).
3. Deploy a replicated-job service with MaxConcurrent: 1 and TotalCompletions: 1.
4. docker inspect the resulting tasks and compare JobIteration.Index, Slot and CreatedAt.

docker service ps alone will not show this clearly — its Name column carries node IDs, not slots.

Expected behavior

One task for the iteration, and the job settling once it completes.

What happens instead is one task per manager that has ever held leadership and has not been restarted since, all carrying the same JobIteration.Index and the same Slot, created milliseconds apart, each running the payload — followed by another such batch as soon as they finish.

docker version

Server 27.3.1 / API 1.47 / Go go1.22.7 / linux-amd64

docker info

containerd 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc       v1.1.14-0-g2c9f560
kernel     6.17.0-1022-azure
os         Ubuntu 24.04.3 LTS
cgroup     systemd
swarm      active, 5 managers, 5 nodes

Host, network and service details omitted; happy to supply more on request.

Additional Info

The bug is in the manager control plane and is not specific to this environment — the relevant code is unchanged on swarmkit master as of today, so current releases are affected too.