Race condition in HA: concurrent ATC instances attempt to create the same container, causing "already exists" error and silent duplicate task execution

Summary

In an HA deployment with multiple ATC (web) instances, a rolling restart can cause two ATCs to concurrently attempt to create the same container on a worker. The second attempt fails with new container: container "X": already exists, erroring the build and triggering a retry — but the original container continues running silently on the worker, causing downstream side effects (e.g. conflicting Terraform state on the retry).

Steps to Reproduce

1. Run Concourse with two web (ATC) instances sharing the same PostgreSQL database
2. Trigger a build that runs a task step
3. While the build is in the container-creation phase (between FindContainer returning state=creating and creatingContainer.Created() being called), perform a rolling restart — stop web1, allowing web2 to acquire the build tracking advisory lock
4. After web2 picks up the build and enters createGardenContainer, restart web1
5. Web1 re-acquires the build tracking lock, calls Lookup (returns nil — container not yet in containerd), and also enters createGardenContainer
6. Both ATCs call gardenClient.Create with the same container handle concurrently

Expected Results

Only one ATC creates the container. The other either attaches to the existing container or waits. The build completes once without duplicate execution.

Actual Results

atc.tracker.tick.run.failed-to-create-container-in-garden
  error: "new container: container \"4330d7d3-941c-4ca1-8dea-5b8175c7284a\": already exists"

atc.tracker.tick.run.errored
  error: "run check: find or create container on worker db634eed7c74:
          new container: container \"4330d7d3-941c-4ca1-8dea-5b8175c7284a\": already exists"

The build is marked as errored and a retry is scheduled on a different worker. The original container continues executing on the first worker unobserved, causing side effects (e.g. Terraform state lock conflicts, duplicate deployments).

Web Node(s) configuration

1. The build tracking advisory lock (pg_try_advisory_lock(1, buildID)) is session-level and releases when the ATC's PostgreSQL connection is closed — this occurs during a rolling restart or when the connection pool recycles an idle connection
2. A second ATC acquires the now-free lock and calls indOrCreateContainer
3. The DB shows state=creating; gardenClient.Lookup returns nil (container not yet in containerd)
4. Both ATCs proceed into createGardenContainer → gardenClient.Create
5. The Garden backend's createLock serializes the two HTTP requests — first succeeds, second calls containerd.NewContainer and receives ErrAlreadyExists
6. The error propagates as "new container: container X: already exists" with no recovery path — markContainerAsFailed is called and the build errors

Local changes for easier reproduce: docker-compose.yml — add a second web service:

web2:
  # copy of web service
  ports: [8081:8080]
  environment:
    CONCOURSE_PEER_ADDRESS: web2
    # ... same as web

Worker CONCOURSE_TSA_HOST: web:2222,web2:2222

atc/worker/gardenruntime/worker.go — sleep before createGardenContainer to widen the race window:

if gardenContainer == nil {
    time.Sleep(60 * time.Second) // gives time to stop/start web instances
    gardenContainer, err = worker.createGardenContainer(...)
}

worker/runtime/backend.go — sleep inside createContainer to hold createLock while the second ATC queues its request:

// before b.client.NewContainer(...)
time.Sleep(60 * time.Second)
return b.client.NewContainer(ctx, gdnSpec.Handle, labels, oci)

Worker(s) configuration

No response

Concourse Version

8.3.0

Browser (if applicable)

No response

Did this use to work?

No response