bug: pending-shutdown lock release uses execWf (empty Status for workflowTemplateRef), leaking the wait-queue entry forever

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself

What happened/what you expected to happen?

Terminating a Workflow while it is Pending, waiting on a workflow-level synchronization mutex/semaphore, permanently leaks that Workflow's entry from the lock's in-memory wait queue when the Workflow was submitted with workflowTemplateRef (i.e. every real-world "submit from WorkflowTemplate" usage). The lock then appears permanently held/blocked (Lock status: N/N, "isn't at the front") with no Running holder anywhere, until the workflow-controller process itself is restarted.

Expected: terminating a Pending, not-yet-started Workflow releases its wait-queue entry the same way the completion path already does for a Workflow that ran.

Reproduction / root cause

releaseLocksForPendingShuttingdownWfs (workflow/controller/operator.go) releases locks with:

if woc.controller.syncManager.ReleaseAll(ctx, woc.execWf) {

For a workflowTemplateRef Workflow, setExecWorkflow (same file) reassigns woc.execWf to a brand-new object:

woc.execWf = &wfv1.Workflow{Spec: *woc.wf.Status.StoredWorkflowSpec.DeepCopy()}

.Status on this new object is the zero value, so execWf.Status.Synchronization is always nil. Manager.ReleaseAll (workflow/sync/sync_manager.go) starts with:

if wf.Status.Synchronization == nil {
    return true
}

so the call returns true (logged as "Released all locks since this pending workflow is being shutdown") without ever calling removeFromQueue for any lock the Workflow was actually queued on. The real object, woc.wf, does carry the populated Status.Synchronization.Mutex.Waiting/Semaphore.Waiting entries (they were written onto it by TryAcquire, which is correctly called with woc.wf a few lines above), but that object is never passed to ReleaseAll here.

The normal completion path in the same file ("Release all acquired lock for completed workflow") calls the correct object:

if woc.controller.syncManager.ReleaseAll(ctx, woc.wf) {

so a Workflow that ran and completed never leaks. Only the pending-shutdown path is affected, and only for workflowTemplateRef Workflows (a bare/inline Workflow has execWf == wf, the same pointer, which is presumably why this was not caught by #16776/#16777's own reproduction — both used inline Workflow specs).

Live confirmation on a production v4.1.2 cluster: a Workflow-level synchronization.mutexes with two named mutexes, four redundant workflowTemplateRef Workflows queued Pending on them, spec.shutdown: Terminate patched onto each. All four correctly transitioned to Failed "Stopped with strategy 'Terminate'" (the #16776 fix worked for that part), but the next Workflow queued on the same mutex logged "isn't at the front" against a holderKey matching one of the four terminated Workflows, indefinitely, until the controller Deployment was restarted (which rebuilds the in-memory queue from live Workflow objects and is the only thing that cleared it).

Suggested fix

releaseLocksForPendingShuttingdownWfs should call syncManager.ReleaseAll(ctx, woc.wf), matching the completion path, not woc.execWf. execWf.Spec is still what should be checked for GetShutdownStrategy()/Synchronization presence elsewhere in this function if needed, but the object passed to ReleaseAll must be the one whose .Status TryAcquire actually populated.

Version

v4.1.2 (confirmed still present, unfixed, on release-4.1 branch HEAD as of 2026-09-09, several commits past the v4.1.2 tag)

Paste a small workflow that reproduces the issue.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: sync-leak-repro
  namespace: argo
spec:
  entrypoint: whalesay
  synchronization:
    mutexes:
      - name: sync-leak-repro
  templates:
    - name: whalesay
      container:
        image: docker/whalesay:latest
        command: [sh, -c]
        args: ["sleep 99999"]
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: sync-leak-repro-holder-
  namespace: argo
spec:
  workflowTemplateRef:
    name: sync-leak-repro
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: sync-leak-repro-waiter-
  namespace: argo
spec:
  workflowTemplateRef:
    name: sync-leak-repro

1. Submit the holder Workflow (acquires the mutex, sleeps).
2. Submit the waiter Workflow (queues Pending, Waiting for argo/Mutex/sync-leak-repro lock).
3. kubectl patch wf <waiter> --type merge -p '{"spec":{"shutdown":"Terminate"}}' — the waiter correctly becomes Failed.
4. Submit a third Workflow against the same WorkflowTemplate while the holder is still running, then terminate the holder. The third Workflow never acquires the lock: it logs "isn't at the front" forever, because the waiter's stale queue entry from step 3 is still there. Only restarting the controller clears it.

Logs from the workflow controller

time="...Z" level=info msg="Released all locks since this pending workflow is being shutdown" key=argo-workflows/build-publish-dating-<name>
time="...Z" level=info msg="isn't at the front" holderKey=argo-workflows/build-publish-dating-<name> lockType=mutex name=argo-workflows/Mutex/build-publish-dating component=workflow_worker workflow=<a later, unrelated workflow>