Helm V4 hangs infinetely on "waiting for resource" when kind=Rollout expectedStatus=Current actualStatus=Unknown
What happened?
When a chart contains an Argo Rollout custom resource (or other CRD), running helm install --wait (or an upgrade with --wait) with Helm v4 hangs: Helm logs
waiting for resource ... kind=Rollout expectedStatus=Current actualStatus=Unknownand keeps polling until the --timeout is reached, even though the Rollout is healthy and fully rolled out. This affects Helm v4 since v4.0.0 (observed on v4.2.4).
What did you expect to happen?
Helm v4's wait logic (pkg/kube/statuswait.go) computes resource readiness with the kstatus library. kstatus has dedicated status readers for well-known built-in kinds (Deployment, StatefulSet, DaemonSet, Pod, Job, PVC, ...). For any other kind it falls back to a generic status reader that evaluates readiness using Kubernetes status conventions:
status.observedGenerationmust be a number that matchesmetadata.generationstatus.conditionsmust follow the standardReady/Availablepatterns
An Argo Rollout does not follow these conventions:
- its
status.observedGenerationis a string, not an integer - its
status.conditionsuse custom types (e.g.Promoted,Progressing) instead of the genericReady/Available
As a result kstatus cannot compute the Rollout's status and reports it as Unknown with an attached error. Helm's status observer only treated resources with status == Current as ready, so the Rollout stayed Unknown forever and the wait hung until the timeout.
A second, closely related problem: the status watcher starts one informer per GroupKind. If the backing CRD is created by the same release (a crds/ directory) and is not yet established when the informer starts, the watcher abandons that informer permanently, again leaving the resource in Unknown until the wait times out.
How can we reproduce it (as minimally and precisely as possible)?
Prerequisites
minikube,kubectl,curlandhelmfileonPATH- a Helm build to test by default the script uses
$(which helm); pass the path to your build as the first positional argument. Build it first withmake build.
Script (minikube-argo-rollout.sh)
#!/usr/bin/env bash
RESET=0
POS_ARGS=()
while [[ $# -gt 0 ]]; do
case "$1" in
--reset|-r)
RESET=1
;;
-h|--help)
echo "usage: $0 [--reset|-r] [HELM_BIN] [RELEASE_NAME] [NAMESPACE]"
echo
echo " --reset|-r delete all resources created by a previous run and exit"
echo " HELM_BIN path to the helm binary (default: what ever is under $(which helm))"
echo " RELEASE_NAME release name (default: app)"
echo " NAMESPACE namespace to install into (default: default)"
exit 0
;;
-*)
echo "error: unknown option: $1" >&2
exit 1
;;
*)
POS_ARGS+=("$1")
;;
esac
shift
done
HELM_BIN="${POS_ARGS[0]:-$(which helm)}"
RELEASE_NAME="${POS_ARGS[1]:-app}"
NAMESPACE="${POS_ARGS[2]:-default}"
CHART_DIR="$(mktemp -d)/chart"
ARGO_NAMESPACE="argo-rollouts"
ARGO_INSTALL_URL="https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml"
say() { printf '\n==> %s\n' "$*"; }
command_exists() { command -v "$1" >/dev/null 2>&1; }
is_protected_namespace() {
case "$1" in
default|kube-system|kube-public|kube-node-lease) return 0 ;;
*) return 1 ;;
esac
}
reset_argo() {
say "uninstalling release '$RELEASE_NAME'"
"$HELM_BIN" uninstall "$RELEASE_NAME" --namespace "$NAMESPACE" >/dev/null 2>&1 || true
say "deleting Argo Rollouts CRDs"
for crd in $(kubectl get crd -o name 2>/dev/null | grep 'argoproj.io'); do
kubectl delete "$crd" --ignore-not-found --wait=false >/dev/null 2>&1 || true
done
say "deleting namespace '$ARGO_NAMESPACE'"
kubectl delete namespace "$ARGO_NAMESPACE" --ignore-not-found --wait=true --timeout=60s >/dev/null 2>&1 || true
if ! is_protected_namespace "$NAMESPACE"; then
say "deleting namespace '$NAMESPACE'"
kubectl delete namespace "$NAMESPACE" --ignore-not-found --wait=true --timeout=60s >/dev/null 2>&1 || true
fi
}
# --- Pre-flight checks -------------------------------------------------------
for cmd in minikube kubectl curl helmfile; do
if ! command_exists "$cmd"; then
echo "error: '$cmd' is required but was not found in PATH" >&2
exit 1
fi
done
if [[ ! -x "$HELM_BIN" ]]; then
echo "error: helm binary not found or not executable: $HELM_BIN" >&2
echo "build it first with: make build (outputs bin/helm)" >&2
exit 1
fi
# --- Minikube ----------------------------------------------------------------
if ! minikube status >/dev/null 2>&1; then
say "starting minikube"
minikube start
fi
kubectl cluster-info >/dev/null 2>&1 || { echo "error: kubectl cannot reach the cluster" >&2; exit 1; }
say "minikube is running"
if [[ "$RESET" -eq 1 ]]; then
reset_argo
say "reset complete"
exit 0
fi
# --- Argo Rollouts -----------------------------------------------------------
if ! kubectl get crd rollouts.argoproj.io >/dev/null 2>&1; then
say "installing Argo Rollouts"
kubectl create namespace "$ARGO_NAMESPACE" >/dev/null 2>&1 || true
manifest="$(mktemp)"
crd_manifest="$(mktemp)"
rest_manifest="$(mktemp)"
trap 'rm -f "$manifest" "$crd_manifest" "$rest_manifest"' EXIT
curl -sSL "$ARGO_INSTALL_URL" -o "$manifest"
# Split the CRDs from the controller manifest. CRDs must be installed with
# `kubectl create`: `kubectl apply` writes the full manifest into the
# last-applied-configuration annotation, which exceeds the 256 KiB annotation
# limit for the large CRD schemas shipped in current Argo Rollouts releases.
# The awk re-inserts the `---` YAML document separators.
awk -v crdfile="$crd_manifest" -v restfile="$rest_manifest" \
'BEGIN{RS="\n---\n"} {f=($0 ~ /kind: CustomResourceDefinition/)? crdfile : restfile; if (seen[f]++) print "---" > f; print $0 > f}' \
"$manifest"
kubectl create -f "$crd_manifest" || {
echo "error: failed to create Argo Rollouts CRDs" >&2
exit 1
}
kubectl apply -n "$ARGO_NAMESPACE" -f "$rest_manifest"
else
say "Argo Rollouts CRD already present, skipping install"
fi
say "waiting for the Rollout CRD to be established"
kubectl wait --for=condition=Established crd/rollouts.argoproj.io --timeout=120s || {
echo "error: Rollout CRD was not established" >&2
exit 1
}
kubectl get crd rollouts.argoproj.io >/dev/null 2>&1 || {
echo "error: Rollout CRD is not present on the cluster" >&2
exit 1
}
say "waiting for the Argo Rollouts controller to be ready"
kubectl -n "$ARGO_NAMESPACE" wait --for=condition=available --timeout=120s deployment/argo-rollouts
kubectl create namespace "$NAMESPACE" >/dev/null 2>&1 || true
# --- Chart -------------------------------------------------------------------
say "creating a sample chart in $CHART_DIR"
mkdir -p "$CHART_DIR/templates"
cat > "$CHART_DIR/Chart.yaml" <<EOF
apiVersion: v2
name: rollout-test
description: Minimal chart with an Argo Rollout to reproduce helm#32615
version: 0.1.0
EOF
cat > "$CHART_DIR/values.yaml" <<EOF
image:
repository: nginx
tag: "1.25"
replicas: 2
EOF
cat > "$CHART_DIR/templates/rollout.yaml" <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: {{ .Release.Name }}
spec:
replicas: {{ .Values.replicas }}
selector:
matchLabels:
app: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ .Release.Name }}
spec:
containers:
- name: app
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
ports:
- containerPort: 80
strategy:
canary:
steps:
- setWeight: 100
EOF
cat > "$CHART_DIR/templates/service.yaml" <<EOF
apiVersion: v1
kind: Service
metadata:
name: {{ .Release.Name }}
spec:
selector:
app: {{ .Release.Name }}
ports:
- port: 80
targetPort: 80
EOF
HELMFILE_DIR="$(dirname "$CHART_DIR")"
cat > "$HELMFILE_DIR/helmfile.yaml" <<EOF
helmBinary: $HELM_BIN
helmDefaults:
wait: true
atomic: true
timeout: 300
releases:
- name: $RELEASE_NAME
namespace: $NAMESPACE
chart: $CHART_DIR
version: 0.1.0
EOF
# --- Install -----------------------------------------------------------------
say "installing release '$RELEASE_NAME' via helmfile"
# Give the rollout a generous timeout: with the buggy build it will hang and
# time out; with the fixed build it should complete in a few seconds.
helmfile --file "$HELMFILE_DIR/helmfile.yaml" apply
say "install succeeded"
"$HELM_BIN" list --namespace "$NAMESPACE"
kubectl -n "$NAMESPACE" get rollout "$RELEASE_NAME" -o jsonpath='{.status.observedGeneration}{"\n"}'
# --- Modify + sync -----------------------------------------------------------
say "modifying the rollout manifest (adding a label) to trigger a change"
awk '{ print } /^ name: / { print " labels:"; print " app: {{ .Release.Name }}" }' \
"$CHART_DIR/templates/rollout.yaml" > "$CHART_DIR/templates/rollout.yaml.new"
mv "$CHART_DIR/templates/rollout.yaml.new" "$CHART_DIR/templates/rollout.yaml"
say "triggering helmfile sync --debug"
helmfile --file "$HELMFILE_DIR/helmfile.yaml" sync --debug
What the script does
- Starts
minikubeif it is not running. - Installs Argo Rollouts:
- CRDs are installed with
kubectl createbecausekubectl applywrites the full manifest into thelast-applied-configurationannotation, which exceeds the 256 KiB annotation limit for the large CRD schemas shipped in current Argo Rollouts releases. - The controller manifest (RBAC, ConfigMap, Secret, Service, Deployment) is applied with
kubectl apply. - Waits for the
rollouts.argoproj.ioCRD to beEstablishedand the controller deployment to be ready.
- CRDs are installed with
- Generates a minimal chart (
rollout-test) with aRolloutand aService. - Installs the chart via
helmfile apply(helm upgrade --installwith--waitand a 300s timeout) — this is the step that reproduces the bug. - Prints the release and the Rollout's
status.observedGeneration. - Adds a
labelsentry to the chart's Rollout template and runshelmfile sync --debugto exercise the same wait path through an upgrade.
Run
# Default helm binary would be whatever $(which helm) will return
chmod +x ./minikube-argo-rollout.sh
./minikube-argo-rollout.sh
# Or a specific Helm build
./minikube-argo-rollout.sh /path/to/helmIMPORTANT
When the bug occurs, SIGINT seems not respected, you may need to stop the helmfile process via kill -9 <PID>
Expected results
- Buggy Helm (pre-fix, e.g. v4.2.4): the install step hangs in
waiting for resource ... kind=Rollout expectedStatus=Current actualStatus=Unknownand times out after 5 minutes. - Fixed Helm (this repo's
main): the install completes in a few seconds, the rollout is reported ready, andstatus.observedGenerationis printed. The subsequenthelmfile sync --debugshows the upgraded manifest with the added label.
Reset
To delete everything the script created and re-run from a clean state:
./minikube-argo-rollout.sh --resetThis uninstalls the release, deletes all *.argoproj.io CRDs and the argo-rollouts namespace, and removes the release namespace (unless it is a protected one such as default).
Helm version
$ helm version
version.BuildInfo{Version:"v4.2.4", GitCommit:"3900f434fd3ef2b84065dc04508df48f288dba00", GitTreeState:"clean", GoVersion:"go1.26.5", KubeClientVersion:"v1.36"}BUT it happens since v4.0.0
Kubernetes version
$ kubectl version
Client Version: v1.36.2
Kustomize Version: v5.8.1
Server Version: v1.34.9-eks-bca9cf6Source: helm/helm