Referencing the output of a `Failed` step under `continueOn` requeues forever instead of failing (since #15442); the workflow cannot be ended by its deadline or by `shutdown`

Author: JoibelCreated Sep 16, 2026Updated Sep 16, 2026

Pre-requisites

I have double-checked my configuration
I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
I have searched existing issues and could not find a match for this bug
I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Since #15442 (v4.0.0, cherry-picked to v3.7.11 in #15657), an unresolved {{steps.X.outputs.parameters.Y}} or {{tasks.X...}} reference in a step returns ErrRequeue and the workflow is requeued. The intent was to wait for a step that has not finished yet. There is no check that the referenced node is already fulfilled, so a reference that can never resolve makes the workflow reconcile every DEFAULT_REQUEUE_TIME (10s) forever.

The skipped/omitted node case has since been covered (#15841, #15932, #16223 populate declared outputs of skipped nodes into scope). A node that finished Failed or Error without producing any outputs is not covered. That happens whenever a pod is killed before its wait container reports a task result: most commonly when activeDeadlineSeconds fires while the pod is Pending or Running. With continueOn on that step the step group is marked successful, the next step's when (or arguments) references the missing output, and the workflow spins.

Consequences, all confirmed with the workflow below on v4.1.2:

The workflow stays Running forever with an unchanged resourceVersion, logging Failed to resolve references ... error=requeue every 10s.
activeDeadlineSeconds cannot end it. After the deadline the controller only refuses to create new pods and fails retries; nothing marks the workflow itself Failed.
spec.shutdown: Terminate does not end it either. Only deleting the workflow does.
Because it never completes, TTL GC never runs. Each new occurrence is a permanent extra reconcile every 10s.

Before v3.7.11 / v4.0.0 the same situation errored the step group (failed to resolve references) and the workflow failed at the deadline, which was the correct outcome.

Expected: if the referenced node is fulfilled and the variable is still missing, fail the step group (as before #15442) so the workflow completes. Only requeue while the referenced node is not yet fulfilled.

Workaround for template authors: use an expression tag with a fallback, e.g. {{= steps.producer.outputs.parameters.flag ?? 'false' }}, so the tag never fails to resolve. A valueFrom.default on the producer's output does not help here because a pod that never reported a task result has no outputs at all.

Version(s)

v4.1.2 (reproduced), v3.7.17 (seen in the field). The requeue path in workflow/controller/steps.go on main is unchanged, so :latest is affected as well.

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

yaml

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: deadline-requeue-loop-
spec:
  entrypoint: main
  # Short deadline so the producer step is still Pending when it fires.
  activeDeadlineSeconds: 20
  templates:
    - name: main
      steps:
        - - name: producer
            template: producer
            # Tolerate the failure so the step group is still considered successful.
            continueOn:
              failed: true
              error: true
        - - name: consumer
            template: print
            # The status tag always resolves. The output tag never does: a node that was
            # killed by the deadline before running has no outputs, not even the default.
            when: '{{steps.producer.status}} == Succeeded && "{{steps.producer.outputs.parameters.flag}}" == "true"'
            arguments:
              parameters:
                - name: msg
                  value: "producer said {{steps.producer.outputs.parameters.flag}}"

    - name: producer
      retryStrategy:
        limit: "2"
      outputs:
        parameters:
          - name: flag
            valueFrom:
              path: /tmp/flag
              default: "false"
      # Unschedulable on purpose so the pod is still Pending at the deadline and never
      # reports a task result. Any pod killed before its wait container reports behaves the same.
      nodeSelector:
        does-not-exist: "true"
      container:
        image: alpine:3.20
        command: [sh, -c]
        args: ["echo true > /tmp/flag"]

    - name: print
      inputs:
        parameters:
          - name: msg
      container:
        image: alpine:3.20
        command: [echo, "{{inputs.parameters.msg}}"]

Resulting node tree (workflow stays Running indefinitely):

Steps      Running    deadline-requeue-loop-chr5k
StepGroup  Succeeded  [0]
Retry      Failed     producer      retry exceeded workflow deadline 2026-09-15 16:38:13 +0000 UTC   (no outputs)
Pod        Failed     producer(0)   Step exceeded its deadline                                        (no outputs)
StepGroup  Running    [1]

Patching spec.shutdown: Terminate afterwards leaves it Running and still looping.

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

time=2026-09-15T16:38:23.669Z level=INFO msg="Processing workflow" namespace=argo component=workflow_worker workflow=deadline-requeue-loop-chr5k phase=Running resourceVersion=586702
time=2026-09-15T16:38:23.669Z level=ERROR msg="Failed to resolve references" stepName=consumer component=workflow_worker workflow=deadline-requeue-loop-chr5k namespace=argo error=requeue
time=2026-09-15T16:38:33.670Z level=INFO msg="Processing workflow" workflow=deadline-requeue-loop-chr5k phase=Running resourceVersion=586702 namespace=argo component=workflow_worker
time=2026-09-15T16:38:33.671Z level=ERROR msg="Failed to resolve references" component=workflow_worker workflow=deadline-requeue-loop-chr5k namespace=argo stepName=consumer error=requeue
time=2026-09-15T16:38:43.672Z level=INFO msg="Processing workflow" workflow=deadline-requeue-loop-chr5k namespace=argo component=workflow_worker phase=Running resourceVersion=586702
time=2026-09-15T16:38:43.673Z level=ERROR msg="Failed to resolve references" error=requeue workflow=deadline-requeue-loop-chr5k namespace=argo stepName=consumer component=workflow_worker
time=2026-09-15T16:38:53.674Z level=INFO msg="Processing workflow" workflow=deadline-requeue-loop-chr5k phase=Running resourceVersion=586702 namespace=argo component=workflow_worker
time=2026-09-15T16:38:53.674Z level=ERROR msg="Failed to resolve references" component=workflow_worker workflow=deadline-requeue-loop-chr5k namespace=argo stepName=consumer error=requeue
... (repeats every 10s, resourceVersion never changes)

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

(no output: the producer pod never scheduled, which is the point)

Source: argoproj/argo-workflows

View original on GitHub View discussion on GitHub