Swarm stack update reports "deployment failed: task: non-zero exit (137)" from a historical failed task, then rolls back the stack file while Swarm runs the new spec

Before you start please confirm the following.

• Yes, I've searched similar issues on GitHub.
• Yes, I've checked whether this issue is covered in the Portainer documentation.

Problem Description

The following bug report has been drafted by AI, but slightly tweaked and tested by me.

Since 2.44.0 STS / 2.45.0 LTS (#13213, PR #13220 "detect async deploy failures instead of reporting success"), updating a Swarm stack fails whenever one of its services has a failed task in Swarm's task history, for example from a container that was OOM-killed or crashed at some earlier point and was long since restarted by Swarm.

The update itself is applied correctly: Swarm performs the rolling update and every service ends up running the new spec. But Portainer's new post-deploy check reads the old failed task, marks the stack as Error with that task's message:

deployment failed: task: non-zero exit (137)

and then rolls back the stored stack file to the previous version. The stack file in Portainer and the services in Swarm now disagree. The next update, from the UI or the API, starts from the stale file and silently re-deploys the old image tags for the services updated in the "failed" run.

We hit this in CI, where a webhook updates image tags via PUT /api/stacks/{id}. On 2.33.7 the same flow worked for years, because Portainer ran docker stack deploy and returned without checking task state.

Expected Behavior

• The post-deploy check only considers tasks that belong to the deployment it is checking: tasks with desired state running (or TaskList with a desired-state=running filter), or tasks created for the current service spec version. A task that failed before the update started says nothing about the update.
• A failed task should only count as a deploy failure if no task of that slot is running or starting.
• If the services were already updated in Swarm, the stored stack file must not be rolled back. At minimum, the rollback should be visible in the UI and API instead of silent.

Actual Behavior

• Stack status becomes Error with deployment failed: task: non-zero exit (137) (the status code nmber depending on how the historical container died).
• docker service ps shows all new tasks running with the new spec.
• GET /api/stacks/{id}/file returns the previous stack file.

Steps to Reproduce

1. Create a Swarm stack:

   yaml复制

   services:
     app:
       image: alpine:3.20
       command: ["sleep", "infinity"]
       environment:
         ROLL: "0"
       deploy:
         replicas: 2
         update_config:
           order: start-first

2. On the node running the tasks, kill or stop one of the service's containers outside Portainer, as an OOM kill or crash would:

   bash复制

   docker kill $(docker ps -q --filter label=com.docker.swarm.service.name=<stack>_app | head -1)

   Swarm restarts the task. docker service ps --no-trunc <stack>_app now shows a task in state Failed with task: non-zero exit (137). This failed task stays in the history for a long time (default task history limit is 5 per slot).

3. Update the stack with any change that triggers a rolling update, for example bump ROLL to "1":

   bash复制

   curl -X PUT -H "X-API-Key: $API_KEY" -H "Content-Type: application/json" \
     "$PORTAINER_URL/api/stacks/$STACK_ID?endpointId=$ENDPOINT_ID" \
     -d '{"StackFileContent": "<same compose with ROLL: \"1\">", "Env": [], "Prune": false, "PullImage": false}'

4. Poll GET /api/stacks/{id}. Within the 30 s check the stack flips to Status: 4 with DeploymentStatus[-1].Message = "deployment failed: task: non-zero exit (137)".

5. Verify the mismatch:

   • docker service inspect <stack>_app --format '{{.Spec.TaskTemplate.ContainerSpec.Env}}' shows ROLL=1 and docker service ps shows both replicas running.
   • GET /api/stacks/{id}/file still contains ROLL: "0".

In our environment one killed container was enough to reproduce on the very next update.

Portainer logs or screenshots

No response

Portainer version

2.45.0

Portainer Edition

Community Edition (CE)

Platform and Version

Docker Swarm

OS and Architecture

Debian 13, AMD64

Browser

Firefox 155.0.1

What command did you use to deploy Portainer?

Additional Information

The behaviour is also present on develop (getServiceStatus is unchanged there). PR #13235 touches the same function for job services and already filters historical tasks by job iteration, which suggests the same kind of filtering is needed for replicated and global services.

Workaround we use in our deploy webhook: when the stack ends in Error with a non-zero exit message, compare each service's com.docker.stack.image label with the new stack file via /endpoints/{id}/docker/services; if Swarm already runs the new images, send the identical stack file again. Nothing changes in Swarm, the count check passes at once, and the stored file is restored.

The analysis of the Portainer source and the first version of this report were done with AI assistance; the reproduction above was run and confirmed manually on our Swarm.

AI also determined (reasoned?) a root cause:

Root cause

getServiceStatus in pkg/libstack/swarm/swarm.go, called from WaitForStatus, which SwarmStackManager.Deploy (api/exec/swarm_stack.go) now runs for up to 30 s after the deploy:

1. It lists tasks with only a service filter, so the result includes tasks with desired state shutdown and the failed tasks kept in Swarm's history.
2. If the number of running tasks equals the replica count it returns Running. Otherwise it iterates the task list in API order and returns on the first task it sees. A Failed or Rejected task immediately yields StatusError with that task's Status.Err, without considering the other tasks.

During a start-first rolling update there is a window per replica where the new and the old task are both running, so the count check fails and the loop runs. Whether the first task in the list is a historical failed one depends on task ID ordering, so with few failed tasks the problem is intermittent; with more it becomes constant. A service with replicas: 0 skips the count check entirely, so one failed task in its history makes every update of that stack fail.

postDeploy in api/http/handler/stacks/stack_update.go then calls RollbackStackFile on the deploy error, although deployServices had already applied every ServiceUpdate.