Cluster Mesh docs: workloads do not need a restart after a cluster ID/name change, but Hubble server certs do need regenerating

Is there an existing issue for this?

• I have searched the existing issues

Version

equal or higher than v1.19.7 and lower than v1.20.0

What happened?

The cluster mesh setup page says that if you change the cluster id or name on a running cluster you have to restart all workloads.

We planned for such changes on a 1.19.0 cluster that was still on the defaults (id 0, name default) and the plan included a full workload restart because of the mentioned note. While preparing the PR on our side, the maintenance window and looking for possible disruptions I decided to check the code and it suggested the opposite: On restart the agent restores the endpoint with its old identity (RegenerateAfterRestore in pkg/endpoint/restore.go), then RunRestoredMetadataResolver reads the pod labels again from the API. SanitizePodLabels puts the current cluster name into io.cilium.k8s.policy.cluster, UpdateLabels sees the label changed, identityLabelsChanged allocates a new identity and regenerates the endpoint. Apparently this wasn't new, the restore path has run the metadata resolver since 1.9 (daemon/cmd/state.go) and the cluster name has been a pod identity label since 1.10 (pkg/k8s/labels.go, now pkg/k8s/utils/utils.go). So the note has been wrong since it was written in 2021, at least for the case where the name changes. We went ahead and only restarted the agents: every pod got its new identity during the rollout, no pod was restarted!

The one case where a restart could matter is changing the id but keeping the name, and there the note doesn't help either: no label changes, so AllocateIdentity reuses the existing identity for that label set (pkg/allocator/allocator.go, lockedAllocate), old number included, and a restarted pod gets the same one. Only identity GC would eventually give a new number. I haven't tested that case, it's from reading the allocator.

What I think the note should say instead: restart the agents (the chart doesn't roll them on a config change) and the pods get their new identities on their own. What does break while the agents roll is policy: selectors get the local cluster name added by default, so pods with the old identity and pods with the new one can't reach each other's policy-protected pods until the rollout is done. Turning policyDefaultLocalCluster off before the change and back on after avoids that, at the cost of two more agent rollouts. (Or something like that, I can think of better wording/phrasing on the PR)

There's a second thing the page doesn't mention. The Hubble server cert has the cluster name in its CN (*.<cluster-name>.hubble-grpc.cilium.io) and with hubble.tls.auto.method=helm the chart keeps the existing hubble-server-certs secret on upgrade (templates/hubble/tls-helm/server-secret.yaml). After the rename the agents announce <node>.<new-name>.hubble-grpc.cilium.io as TLS server name (pkg/hubble/peer/handler.go) but still serve the old cert, and relay can't connect to any peer. You have to delete that secret before the upgrade, or use the cronJob or certmanager method. We regenerated the certs in the same run so we didn't hit it, but it's in the code.

I'm happy to open a PR for the rst.

How can we reproduce the issue?

1. Cilium 1.19.0 with default cluster.id and cluster.name, identity allocation crd, Hubble enabled with hubble.tls.auto.method=helm, some pods running.
2. helm upgrade with cluster.id=1 and cluster.name=cluster-a. Nothing restarts.
3. kubectl -n kube-system rollout restart ds/cilium.
4. cilium-dbg endpoint list on a restarted node: the pods that were already running now have identities above 65535 and the label k8s:io.cilium.k8s.policy.cluster=cluster-a. No pod restarted.
5. Hubble part: leave hubble-server-certs alone before step 3 and check hubble status through relay afterwards.

Cilium Version

cilium-cli: v0.18.7 compiled with go1.25.0 on darwin/arm64
cilium image (default): v1.18.1
cilium image (stable): v1.20.2
cilium image (running): 1.19.0

Kernel Version

6.8.0-106-generic #106-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar  6 07:58:08 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.34.3

Regression

No, the note has been like this since 2021.

Sysdump

No response

Relevant log output

Anything else?

• The note on main: https://github.com/cilium/cilium/blob/main/Documentation/network/clustermesh/setup.rst#L147-L150
• Same text on 1.19: https://github.com/cilium/cilium/blob/v1.19.0/Documentation/network/clustermesh/clustermesh.rst#L152-L155 (the file was renamed to setup.rst on main).
• Code at v1.19.0: pkg/endpoint/restore.go (RegenerateAfterRestore, restoreIdentity), pkg/endpoint/endpoint.go (metadataResolver, identityLabelsChanged), pkg/k8s/utils/utils.go (SanitizePodLabels), pkg/allocator/allocator.go (lockedAllocate), install/kubernetes/cilium/templates/hubble/tls-helm/server-secret.yaml, pkg/hubble/peer/handler.go (TLSServerName).

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct