UI fails to start with error: Failed to load version/info Error: Internal Server Error

Author: jbrinnandCreated May 6, 2025Updated Sep 16, 2026

Labelsgood first issuearea/docs

Pre-requisites

I have double-checked my configuration
I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
I have searched existing issues and could not find a match for this bug
I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Argo workflow version 3.6.7 The argo-server returns appears to get internal server errors when the gui tries to come up. I have tried to access it using port forwarding: kubectl -n argo port-forward service/argo-server 2746:2746 &

Here is what happens when go to: `https://localhost:2746/`

Failed to load version/info Error: Internal Server Error

If I try to access the workflows in the gui this is what appears.

Internal Server Error: header key "cookie" contains value with non-printable ASCII characters

The CLI on which the port forward command was executed also displays the following errors:

E0506 16:41:22.168130 44760 portforward.go:345] error creating error stream for port 2746 -> 2746: Timeout occurred E0506 16:41:22.168227 44760 portforward.go:345] error creating error stream for port 2746 -> 2746: Timeout occurred

The argo-server service logs display the following:

time="2025-05-06T23:43:21.495Z" level=info duration="366.812µs" method=GET path=/api/v1/workflows/ size=96 status=500 │ │ time="2025-05-06T23:43:26.864Z" level=info duration="260.665µs" method=GET path=/api/v1/info size=96 status=500 │ │ time="2025-05-06T23:43:27.455Z" level=info duration="224.239µs" method=GET path=index.html size=487 status=0

The argo-server (pod) logs display the following:

│ time="2025-05-06T23:45:59.616Z" level=info duration="237.498µs" method=GET path=/api/v1/info size=96 status=500 │ │ time="2025-05-06T23:45:59.617Z" level=info duration="368.386µs" method=GET path=/api/v1/userinfo size=96 status=500

The argo cli is at version 3.6.7. It is a binary that I retrieved from the latest release 3.6.7

Version(s)

v3.6,7

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

yaml

kubectl port-forward service/argo-server 2746:2746 &
https://localhost:2746

Note: creating workflows through the command line works. Please see logs below. This looks like some kind of bug with argo-server. It is getting internal server errors.

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

time="2025-05-06T23:24:23.544Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2025-05-06T23:24:23.544Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=0 workflow=webhook-wt47x
time="2025-05-06T23:24:23.544Z" level=info msg="Updated phase  -> Running" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.544Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.545Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.545Z" level=info msg="Pod node webhook-wt47x initialized Pending" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.545Z" level=warning msg="couldn't get boundaryTemplate through nodeName webhook-wt47x" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.568Z" level=info msg="Created pod: webhook-wt47x (webhook-wt47x)" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.568Z" level=info msg="TaskSet Reconciliation" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.568Z" level=info msg=reconcileAgentPod namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.569Z" level=info msg="Workflow to be dehydrated" Workflow Size=1119
time="2025-05-06T23:24:23Z" level=info msg="add pod event" pod=webhook-wt47x
time="2025-05-06T23:24:23Z" level=info msg="update pod event" pod=webhook-wt47x
time="2025-05-06T23:24:23.601Z" level=info msg="Workflow update successful" namespace=argo-events phase=Running resourceVersion=16930401 workflow=webhook-wt47x
time="2025-05-06T23:24:23.604Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=16930401 namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.604Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=0 workflow=webhook-wt47x
time="2025-05-06T23:24:23.605Z" level=info msg="node changed" namespace=argo-events new.message= new.phase=Pending new.progress=0/1 nodeID=webhook-wt47x old.message= old.phase=Pending old.progress=0/1 workflow=webhook-wt47x
time="2025-05-06T23:24:23.605Z" level=info msg="TaskSet Reconciliation" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.606Z" level=info msg=reconcileAgentPod namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:23.606Z" level=info msg="Workflow to be dehydrated" Workflow Size=1423
time="2025-05-06T23:24:23.620Z" level=info msg="Workflow update successful" namespace=argo-events phase=Running resourceVersion=16930403 workflow=webhook-wt47x
time="2025-05-06T23:24:23Z" level=info msg="update pod event" pod=webhook-wt47x
time="2025-05-06T23:24:25Z" level=info msg="update pod event" pod=webhook-wt47x
time="2025-05-06T23:24:30Z" level=info msg="update pod event" pod=webhook-wt47x
time="2025-05-06T23:24:31Z" level=info msg="update pod event" pod=webhook-wt47x
time="2025-05-06T23:24:33Z" level=info msg="update pod event" pod=webhook-wt47x
time="2025-05-06T23:24:33.572Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=16930403 namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:33.573Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=1 workflow=webhook-wt47x
time="2025-05-06T23:24:33Z" level=info msg="queueing pod for cleanup" action=terminateContainers namespace=argo-events podName=webhook-wt47x
time="2025-05-06T23:24:33.573Z" level=info msg="node changed" namespace=argo-events new.message= new.phase=Running new.progress=0/1 nodeID=webhook-wt47x old.message= old.phase=Pending old.progress=0/1 workflow=webhook-wt47x
time="2025-05-06T23:24:33.574Z" level=info msg="TaskSet Reconciliation" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:33.574Z" level=info msg=reconcileAgentPod namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:33.574Z" level=info msg="Workflow to be dehydrated" Workflow Size=1447
time="2025-05-06T23:24:33Z" level=info msg="cleaning up pod" action=terminateContainers key=argo-events/webhook-wt47x/terminateContainers namespace=argo-events podName=webhook-wt47x
time="2025-05-06T23:24:33Z" level=info msg="queueing pod for cleanup after" action=killContainers after=30s namespace=argo-events podName=webhook-wt47x
time="2025-05-06T23:24:33.594Z" level=info msg="Workflow update successful" namespace=argo-events phase=Running resourceVersion=16930466 workflow=webhook-wt47x
time="2025-05-06T23:24:34Z" level=info msg="update pod event" pod=webhook-wt47x
time="2025-05-06T23:24:44.607Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=16930466 namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:44.608Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=1 workflow=webhook-wt47x
time="2025-05-06T23:24:44.609Z" level=info msg="node changed" namespace=argo-events new.message= new.phase=Succeeded new.progress=0/1 nodeID=webhook-wt47x old.message= old.phase=Running old.progress=0/1 workflow=webhook-wt47x
time="2025-05-06T23:24:44.610Z" level=info msg="TaskSet Reconciliation" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:44.610Z" level=info msg=reconcileAgentPod namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:44.610Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:44.611Z" level=info msg="Marking workflow completed" namespace=argo-events workflow=webhook-wt47x
time="2025-05-06T23:24:44.611Z" level=info msg="Workflow to be dehydrated" Workflow Size=1534
time="2025-05-06T23:24:44.631Z" level=info msg="Workflow update successful" namespace=argo-events phase=Succeeded resourceVersion=16930512 workflow=webhook-wt47x
time="2025-05-06T23:24:44Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=argo-events podName=webhook-wt47x
time="2025-05-06T23:24:44Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-events/webhook-wt47x/labelPodCompleted namespace=argo-events podName=webhook-wt47x
time="2025-05-06T23:24:44Z" level=info msg="delete pod event" pod=webhook-wt47x
time="2025-05-06T23:24:45.536Z" level=info msg="Processing workflow" Phase= ResourceVersion=16930519 namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:45.546Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2025-05-06T23:24:45.546Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=0 workflow=webhook-9gjrd
time="2025-05-06T23:24:45.546Z" level=info msg="Updated phase  -> Running" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:45.547Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:45.547Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:45.547Z" level=info msg="Pod node webhook-9gjrd initialized Pending" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:45.547Z" level=warning msg="couldn't get boundaryTemplate through nodeName webhook-9gjrd" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:45.565Z" level=info msg="Created pod: webhook-9gjrd (webhook-9gjrd)" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:45.566Z" level=info msg="TaskSet Reconciliation" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:45.566Z" level=info msg=reconcileAgentPod namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:45.566Z" level=info msg="Workflow to be dehydrated" Workflow Size=1119
time="2025-05-06T23:24:45Z" level=info msg="add pod event" pod=webhook-9gjrd
time="2025-05-06T23:24:45.588Z" level=info msg="Workflow update successful" namespace=argo-events phase=Running resourceVersion=16930522 workflow=webhook-9gjrd
time="2025-05-06T23:24:45Z" level=info msg="update pod event" pod=webhook-9gjrd
time="2025-05-06T23:24:45Z" level=info msg="update pod event" pod=webhook-9gjrd
time="2025-05-06T23:24:47Z" level=info msg="update pod event" pod=webhook-9gjrd
time="2025-05-06T23:24:52Z" level=info msg="update pod event" pod=webhook-9gjrd
time="2025-05-06T23:24:53Z" level=info msg="update pod event" pod=webhook-9gjrd
time="2025-05-06T23:24:55Z" level=info msg="update pod event" pod=webhook-9gjrd
time="2025-05-06T23:24:55.570Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=16930522 namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:55.571Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=1 workflow=webhook-9gjrd
time="2025-05-06T23:24:55Z" level=info msg="queueing pod for cleanup" action=terminateContainers namespace=argo-events podName=webhook-9gjrd
time="2025-05-06T23:24:55.571Z" level=info msg="node changed" namespace=argo-events new.message= new.phase=Running new.progress=0/1 nodeID=webhook-9gjrd old.message= old.phase=Pending old.progress=0/1 workflow=webhook-9gjrd
time="2025-05-06T23:24:55.572Z" level=info msg="TaskSet Reconciliation" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:55.572Z" level=info msg=reconcileAgentPod namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:24:55.572Z" level=info msg="Workflow to be dehydrated" Workflow Size=1447
time="2025-05-06T23:24:55Z" level=info msg="cleaning up pod" action=terminateContainers key=argo-events/webhook-9gjrd/terminateContainers namespace=argo-events podName=webhook-9gjrd
time="2025-05-06T23:24:55Z" level=info msg="queueing pod for cleanup after" action=killContainers after=30s namespace=argo-events podName=webhook-9gjrd
time="2025-05-06T23:24:55.589Z" level=info msg="Workflow update successful" namespace=argo-events phase=Running resourceVersion=16930577 workflow=webhook-9gjrd
time="2025-05-06T23:24:56Z" level=info msg="update pod event" pod=webhook-9gjrd
time="2025-05-06T23:25:03Z" level=info msg="cleaning up pod" action=killContainers key=argo-events/webhook-wt47x/killContainers namespace=argo-events podName=webhook-wt47x
time="2025-05-06T23:25:06.880Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=16930577 namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:25:06.881Z" level=info msg="Task-result reconciliation" namespace=argo-events numObjs=1 workflow=webhook-9gjrd
time="2025-05-06T23:25:06.882Z" level=info msg="node changed" namespace=argo-events new.message= new.phase=Succeeded new.progress=0/1 nodeID=webhook-9gjrd old.message= old.phase=Running old.progress=0/1 workflow=webhook-9gjrd
time="2025-05-06T23:25:06.882Z" level=info msg="TaskSet Reconciliation" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:25:06.882Z" level=info msg=reconcileAgentPod namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:25:06.882Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:25:06.883Z" level=info msg="Marking workflow completed" namespace=argo-events workflow=webhook-9gjrd
time="2025-05-06T23:25:06.883Z" level=info msg="Workflow to be dehydrated" Workflow Size=1534
time="2025-05-06T23:25:06.913Z" level=info msg="Workflow update successful" namespace=argo-events phase=Succeeded resourceVersion=16930624 workflow=webhook-9gjrd
time="2025-05-06T23:25:06Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=argo-events podName=webhook-9gjrd
time="2025-05-06T23:25:06Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo-events/webhook-9gjrd/labelPodCompleted namespace=argo-events podName=webhook-9gjrd
time="2025-05-06T23:25:07Z" level=info msg="delete pod event" pod=webhook-9gjrd
time="2025-05-06T23:25:25Z" level=info msg="cleaning up pod" action=killContainers key=argo-events/webhook-9gjrd/killContainers namespace=argo-events podName=webhook-9gjrd
time="2025-05-06T23:26:28.855Z" level=info msg="Alloc=7230 TotalAlloc=586271 Sys=26965 NumGC=681 Goroutines=187"
time="2025-05-06T23:31:28.855Z" level=info msg="Alloc=6746 TotalAlloc=588033 Sys=26965 NumGC=684 Goroutines=187"
time="2025-05-06T23:36:28.855Z" level=info msg="Alloc=7167 TotalAlloc=590536 Sys=26965 NumGC=686 Goroutines=187"
time="2025-05-06T23:41:28.856Z" level=info msg="Alloc=6814 TotalAlloc=592380 Sys=26965 NumGC=689 Goroutines=187"
time="2025-05-06T23:46:28.855Z" level=info msg="Alloc=7119 TotalAlloc=594214 Sys=26965 NumGC=691 Goroutines=187"
time="2025-05-06T23:51:28.854Z" level=info msg="Alloc=6767 TotalAlloc=596126 Sys=26965 NumGC=694 Goroutines=187"

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Source: argoproj/argo-workflows

View original on GitHub View discussion on GitHub