[BUG] Imported cluster permanently stuck at Connected=False

Author: ChrisMcKeeCreated Sep 17, 2026Updated Sep 17, 2026

Labelskind/bug

Imported cluster permanently stuck at `Connected=False`: `/v3/import/` manifest ships an agent with no `CATTLE_FEATURES`, and the `clusterdeploy` repair phase never runs

This https://github.com/rancher/rancher/issues/55697 looks very similar. We've been running this cluster a while and uprading, so it's not a fresh rancher setup.

Rancher Server Setup

Rancher version: v2.15.1
Installation option: Helm chart on a dedicated EKS cluster (3 replicas), eu-west-2
Proxy/Cert details: AWS Classic ELB terminating TLS with a public ACM certificate (external cert source, cacerts setting empty)

Information about the Cluster

Kubernetes version (downstream): EKS v1.35.8, Bottlerocket nodes, eu-west-1
Cluster Type: Imported (registered), created via the UI "Import Existing → Generic" flow
Network: downstream VPC peered to the Rancher VPC. Other imported clusters on the same Rancher, same topology, are Active and healthy.

User Information

Role: Admin (global admin; downstream cattle SA verified to have cattle-admin with */*)

Describe the bug

A freshly imported cluster connects its agent tunnel successfully but never becomes Active. It sits at Connected=False / Ready=False (Disconnected) / agentDeployed=false forever, with no error surfaced in the UI, in cattle-cluster-agent logs, or in Rancher logs (even at debug).

Reading the source, cluster registration is a two-phase bootstrap:

Phase	Who	Renders `CATTLE_FEATURES`?
1. Import manifest served by `/v3/import/{token}_{clusterID}.yaml`	`ClusterImportHandler`	❌ No hardcodes `AgentFeatures: nil`
2. Agent re-deployment	`clusterdeploy` controller	✅ Yes `systemtemplate.GetDesiredFeatures(cluster)`

The phase-1 agent is incapable of ever satisfying the Connected condition, because multi-cluster-management-agent defaults to false. Registration therefore depends entirely on phase 2 running. When phase 2 stalls for any of its several silent early-returns, the cluster is bricked with zero diagnostics.

Manually adding the env var that phase 2 would have added fixes it instantly and permanently.

To Reproduce

Rancher 2.15.1 behind an external TLS terminator with a publicly-trusted cert.
UI → Import Existing → Generic, then run the generated kubectl apply -f https://<rancher>/v3/import/<token>_<clusterID>.yaml on a fresh EKS cluster.
The agent starts and connects cleanly:

log

INFO: https://<rancher>/ping is accessible
time="..." level=info msg="Rancher agent version v2.15.1 is starting"
time="..." level=info msg="Connecting to wss://<rancher>/v3/connect/register with token starting with prrx..."
time="..." level=info msg="Connected to proxy" url="wss://<rancher>/v3/connect/register"
time="..." level=info msg="DesiredSet - No change(2) /v1, Kind=Secret cattle-system/stv-aggregation for rancher-stv-aggregation"

Rancher logs show the tunnel and cluster controllers coming up:

log

[INFO] Handling backend connection request [c-zfrkc]
[INFO] Starting cluster controllers for c-zfrkc
[INFO] Starting cluster agent for c-zfrkc [owner=true]

…and then nothing else, forever.

The served manifest, verbatim

This is the complete cluster-register container spec returned by /v3/import/. Note the env: block no CATTLE_FEATURES:

yaml

      containers:
        - name: cluster-register
          imagePullPolicy: IfNotPresent
          env:
          - name: CATTLE_SERVER
            value: "https://<rancher>"
          - name: CATTLE_CA_CHECKSUM
            value: ""
          - name: CATTLE_CLUSTER
            value: "true"
          - name: CATTLE_K8S_MANAGED
            value: "true"
          - name: CATTLE_SYSTEM_DEFAULT_REGISTRY
            value: "<acct>.dkr.ecr.eu-west-2.amazonaws.com/docker-hub"
          - name: CATTLE_CREDENTIAL_NAME
            value: cattle-credentials-d8a6162101
          - name: CATTLE_SUC_APP_NAME_OVERRIDE
            value: ""
          - name: CATTLE_SERVER_VERSION
            value: v2.15.1
          - name: CATTLE_INSTALL_UUID
            value: a5ac3907-...
          - name: CATTLE_INGRESS_IP_DOMAIN
            value: sslip.io
          - name: STRICT_VERIFY
            value: "false"
          image: "<acct>.dkr.ecr.eu-west-2.amazonaws.com/docker-hub/rancher/rancher-agent:v2.15.1"

Confirmed live on the cluster:

bash

$ kubectl -n cattle-system get deploy cattle-cluster-agent \
    -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="CATTLE_FEATURES")].value}{"\n"}'

→ empty (40+ minutes after import; phase 2 never ran).

A working cluster on the same Rancher, for comparison:

bash

$ kubectl -n cattle-system get deploy cattle-cluster-agent \
    -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="CATTLE_FEATURES")].value}{"\n"}'
fleet=false,managed-system-upgrade-controller=false,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningprebootstrap=false,provisioningv2=false,rke2=false,turtles=false,ui-sql-cache=true

That string is exactly the 9 keys from GetDesiredFeatures(), alphabetically sorted by toFeatureString() i.e. proof the working cluster went through phase 2 and the broken one did not.

Result

yaml

status:
  conditions:
  - type: AgentTlsStrictCheck
    status: "False"
  - type: Connected
    status: "False"
  - type: Ready
    status: "False"
    reason: Disconnected
  agentDeployed: false

Rancher at debug only ever repeats:

[DEBUG] Skip updating cluster condition ready - cluster agent for [c-zfrkc] isn't connected yet

A healthy cluster appears to register two remotedialer sessions (one being STV); the broken one only ever registers the first:

# broken cluster three reconnects, never an stv-cluster session
[INFO] Handling backend connection request [c-zfrkc]
[INFO] Handling backend connection request [c-zfrkc]
[INFO] Handling backend connection request [c-zfrkc]

# healthy cluster
[INFO] Handling backend connection request [c-xwhrp]
[INFO] Handling backend connection request [stv-cluster-c-xwhrp]

Downstream symptoms: cattle-fleet-system never populated, no fleet-agent, no rancher-webhook, no clusters.fleet.cattle.io object, cluster invisible in the UI.

Expected Result

The cluster registers. Failing that, the failure is visible a condition message or a log line saying why would be nice and save a lot of spelunking.

Workaround

Diffed the deployments and applying CATTLE_FEATURES immediately allowed the agent to complete its registration and wake up.

bash

kubectl -n cattle-system set env deploy/cattle-cluster-agent \
  CATTLE_FEATURES="fleet=false,managed-system-upgrade-controller=false,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningprebootstrap=false,provisioningv2=false,rke2=false,turtles=false,ui-sql-cache=true"

Generated Root cause analysis (mileage may vary)

All line references verified against tag v2.15.1.

1. The import handler explicitly passes `nil` features

pkg/api/norman/customization/clusterregistrationtokens/import.go (ClusterImportHandler):

ops := &systemtemplate.TemplateOps{
 AgentImage:     agentImage,
 AuthImage:      authImage,
 Namespace:      "",
 Token:          token,
 URL:            url,
 IsPreBootstrap: false,
 Cluster:        cluster,          // <-- the cluster object IS available here
 AgentFeatures:  nil,              // <-- but features are hardcoded nil
 Taints:         nil,
 SecretLister:   ch.SecretLister,
 PcExists:       false,
 Mutator:        namespace.GetMutator(),
}
if err = systemtemplate.SystemTemplate(resp, ops); err != nil {

(Long-standing: the same positional nil for agentFeatures is present in release/v2.11 through release/v2.14, and on main.)

2. `nil` renders to an empty string, which the template silently drops

pkg/systemtemplate/import.go:

func toFeatureString(features map[string]bool) string {
 buf := &strings.Builder{}
 var keys []string
 for k := range features { keys = append(keys, k) }
 sort.Strings(keys)
 ...
 return buf.String()      // nil map -> ""
}
...
context := &clusterAgentContext{
 Features: toFeatureString(ops.AgentFeatures),
 ...
}

pkg/systemtemplate/template.go:197-201:

gotemplate

          env:
          {{- if ne .Features "" }}
          - name: CATTLE_FEATURES
            value: "{{.Features}}"
          {{- end }}

So the env var is omitted entirely, with no warning.

3. `multi-cluster-management-agent` defaults to `false`

pkg/features/feature_gates.go:

MCMAgent = newFeature("multi-cluster-management-agent", ..., false, ...)

This feature is agent-only it does not exist as a features.management.cattle.io CR on the local cluster, so an operator cannot see it, list it, or set it:

bash

$ kubectl get features.management.cattle.io multi-cluster-management-agent
Error from server (NotFound): features.management.cattle.io "multi-cluster-management-agent" not found

4. Without it, the agent never starts the embedded steve server

pkg/agent/rancher/rancher.go → Run() returns early when !features.MCMAgent.Enabled(). Silently there is no log line for this. (It also returns early if a rancher Service exists in downstream cattle-system; not our case, verified services "rancher" not found.)

5. `Connected` is derived solely from the steve tunnel

pkg/controllers/management/clusterconnected/clusterconnected.go:

clientKey := proxy.Prefix + cluster.Name   // "stv-cluster-" + cluster.Name
hasSession := c.tunnelServer.HasSession(clientKey)

Polled on a 15s ticker. The ordinary c-<id> agent tunnel is a different session and does not count. So a fully-connected phase-1 agent yields Connected=False forever.

6. Which blocks Ready, which is what the user sees

pkg/controllers/managementuser/healthsyncer/healthsyncer.go bails while clusterconnected.Connected.IsFalse(cluster), emitting the isn't connected yet line and leaving agentDeployed=false.

7. Phase 2 the only thing that can break the cycle has four silent early-returns

pkg/controllers/management/clusterdeploy/clusterdeploy.go:

func (cd *clusterDeploy) doSync(cluster *apimgmtv3.Cluster) error {
 if !apimgmtv3.ClusterConditionProvisioned.IsTrue(cluster) {
  logrus.Tracef(...)                 // (1) trace-level only
  return nil
 }
 uc, err := cd.clusterManager.UserContextNoControllersReconnecting(cluster.Name, false)
 if err != nil { return err }           // (2)
 if err := healthsyncer.IsAPIUp(...); err != nil {
  logrus.Tracef(...)                 // (3) trace-level only
  return ErrCantConnectToAPI
 }
 nodes, err := cd.nodeLister.List(cluster.Name, labels.Everything())
 if err != nil { return err }
 if len(nodes) == 0 {
  return nil                          // (4) NO log at all
 }
 ...
 err = cd.deployAgent(cluster)           // <-- the only writer of CATTLE_FEATURES

Note redeployAgent() would definitely have fired for this cluster it short-circuits to true on !ClusterConditionAgentDeployed.IsTrue(cluster), and agentFeaturesChanged(desired, cluster.Status.AgentFeatures) is trivially true when Status.AgentFeatures is empty and GetDesiredFeatures returns 9 keys. So doSync must have returned at one of (1)–(4) all of which are invisible at debug level (three are Tracef, one logs nothing).

Summary

Phase 1 deliberately ships an agent that cannot satisfy Connected. Phase 2 is the sole repair mechanism, is gated behind four conditions, and reports nothing when it declines to run. The result is an unrecoverable, undiagnosable registration hang.

Requested fixes

One-line fix: stop passing nil. ClusterImportHandler already holds the *apimgmtv3.Cluster. Change AgentFeatures: nil → AgentFeatures: systemtemplate.GetDesiredFeatures(cluster). The import manifest would then bootstrap a fully-functional agent and the two-phase ordering dependency disappears. If phase 1 must stay minimal, at minimum force multi-cluster-management-agent=true, since nothing works without it.
Log the agent's silent exit. pkg/agent/rancher/rancher.go should log at info when it returns early due to !features.MCMAgent.Enabled(). This single line would have reduced a multi-hour investigation to minutes.
Make clusterdeploy's early-returns visible. Promote (1)/(3) to debug and add a log to the len(nodes) == 0 return. A controller that is the sole repair path for a bricked cluster should not decline to run silently.
Surface the real reason on the cluster object. When a c-<id> session exists but stv-cluster-<id> does not after ~N seconds, set a Connected condition message such as: "agent tunnel connected but embedded steve server is not running check CATTLE_FEATURES / multi-cluster-management-agent on the cattle-cluster-agent deployment". Currently Connected=False carries no message at all.

Import yaml file (with redacted values) ```yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: proxy-clusterrole-kubeapiserver rules: - apiGroups: [""] resources: - nodes/metrics - nodes/proxy - nodes/stats - nodes/log - nodes/spec verbs: ["get", "list", "watch", "create"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: proxy-role-binding-kubernetes-master roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: proxy-clusterrole-kubeapiserver subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kube-apiserver --- apiVersion: v1 kind: Namespace metadata: name: cattle-system ---

apiVersion: v1 kind: ServiceAccount metadata: name: cattle namespace: cattle-system

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: cattle-admin-binding namespace: cattle-system labels: cattle.io/creator: "norman" subjects:

kind: ServiceAccount name: cattle namespace: cattle-system roleRef: kind: ClusterRole name: cattle-admin apiGroup: rbac.authorization.k8s.io

apiVersion: v1 kind: Secret metadata: name: cattle-credentials-060e42106f namespace: cattle-system type: Opaque data: url: "" token: "" namespace: ""

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: cattle-admin labels: cattle.io/creator: "norman" rules:

apiGroups:
- '*' resources:
- '*' verbs:
- '*'
nonResourceURLs:
- '*' verbs:
- '*'

apiVersion: apps/v1 kind: Deployment metadata: name: cattle-cluster-agent namespace: cattle-system annotations: management.cattle.io/scale-available: "2" spec: selector: matchLabels: app: cattle-cluster-agent template: metadata: labels: app: cattle-cluster-agent spec: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - preference: matchExpressions: - key: node-role.kubernetes.io/controlplane operator: In values: - "true" weight: 100 - preference: matchExpressions: - key: node-role.kubernetes.io/control-plane operator: In values: - "true" weight: 100 - preference: matchExpressions: - key: cattle.io/cluster-agent operator: In values: - "true" weight: 1 requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: NotIn values: - windows podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - cattle-cluster-agent topologyKey: kubernetes.io/hostname weight: 100 serviceAccountName: cattle tolerations: # No taints or no controlplane nodes found, added defaults - effect: NoSchedule key: node-role.kubernetes.io/controlplane value: "true" - effect: NoSchedule key: "node-role.kubernetes.io/control-plane" operator: "Exists" containers: - name: cluster-register imagePullPolicy: IfNotPresent env: - name: CATTLE_SERVER value: "https://" - name: CATTLE_CA_CHECKSUM value: "" - name: CATTLE_CLUSTER value: "true" - name: CATTLE_K8S_MANAGED value: "true" - name: CATTLE_SYSTEM_DEFAULT_REGISTRY value: ".dkr.ecr.eu-west-2.amazonaws.com/docker-hub" - name: CATTLE_CREDENTIAL_NAME value: cattle-credentials-060e42106f - name: CATTLE_SUC_APP_NAME_OVERRIDE value: "" - name: CATTLE_SERVER_VERSION value: v2.15.1 - name: CATTLE_INSTALL_UUID value: - name: CATTLE_INGRESS_IP_DOMAIN value: sslip.io - name: STRICT_VERIFY value: "false" image: ".dkr.ecr.eu-west-2.amazonaws.com/docker-hub/rancher/rancher-agent:v2.15.1" volumeMounts: - name: cattle-credentials mountPath: /cattle-credentials readOnly: true volumes: - name: cattle-credentials secret: secretName: cattle-credentials-060e42106f defaultMode: 320 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 0 maxSurge: 1

apiVersion: v1 kind: Service metadata: name: cattle-cluster-agent namespace: cattle-system spec: ports:

port: 80 targetPort: 80 protocol: TCP name: http
port: 443 targetPort: 444 protocol: TCP name: https-internal selector: app: cattle-cluster-agent

</details>

Source: rancher/rancher

View original on GitHub View discussion on GitHub

[BUG] Imported cluster permanently stuck at Connected=False

Imported cluster permanently stuck at Connected=False: /v3/import/ manifest ships an agent with no CATTLE_FEATURES, and the clusterdeploy repair phase never runs

Rancher Server Setup

Information about the Cluster

User Information

Describe the bug

To Reproduce

The served manifest, verbatim

Result

Expected Result

Workaround

1. The import handler explicitly passes nil features

2. nil renders to an empty string, which the template silently drops

3. multi-cluster-management-agent defaults to false

4. Without it, the agent never starts the embedded steve server

5. Connected is derived solely from the steve tunnel

6. Which blocks Ready, which is what the user sees

7. Phase 2 the only thing that can break the cycle has four silent early-returns

Summary

Requested fixes

Imported cluster permanently stuck at `Connected=False`: `/v3/import/` manifest ships an agent with no `CATTLE_FEATURES`, and the `clusterdeploy` repair phase never runs

1. The import handler explicitly passes `nil` features

2. `nil` renders to an empty string, which the template silently drops

3. `multi-cluster-management-agent` defaults to `false`

5. `Connected` is derived solely from the steve tunnel