panic: close of closed channel in kubeController.Close() during remote cluster credential rotation

Summary

A panic: close of closed channel occurs in kubeController.Close() when a remote cluster's kubeconfig secret is updated twice in rapid succession (e.g. automated credential rotation). The panic is a double-close of the kubeController.stop channel.

Stack Trace

panic: close of closed channel

goroutine XXXXXXX [running]:
istio.io/istio/pilot/pkg/serviceregistry/kube/controller.(*kubeController).Close(...)
        istio.io/istio/pilot/pkg/serviceregistry/kube/controller/multicluster.go:59 +0x2b
istio.io/istio/pkg/kube/multicluster.(*Component[...]).clusterUpdated(...)
        istio.io/istio/pkg/kube/multicluster/component.go:72 +0xdf
istio.io/istio/pkg/kube/multicluster.(*Cluster).Run(...)
        istio.io/istio/pkg/kube/multicluster/cluster.go:113 +0x2fb
istio.io/istio/pkg/kube/multicluster.(*Controller).addSecret.func1()
        istio.io/istio/pkg/kube/multicluster/secretcontroller.go:385 +0x735
created by istio.io/istio/pkg/kube/multicluster.(*Controller).addSecret
        istio.io/istio/pkg/kube/multicluster/secretcontroller.go:384 +0x735

Root Cause

The race is in Component.clusterUpdated() in pkg/kube/multicluster/component.go as it exists on release-1.29:

func (m *Component[T]) clusterUpdated(cluster *Cluster) ComponentConstraint {
    // Build outside of the lock, in case its slow
    comp := m.constructor(cluster)       // no lock held; can take seconds
    old, f := m.clusters[cluster.ID]    // ← MAP READ WITH NO LOCK
    m.mu.Lock()
    m.clusters[cluster.ID] = comp
    m.mu.Unlock()
    if f {
        old.Close()
    }
    return comp
}

m.clusters[cluster.ID] is read without holding any lock, after a constructor call that can be slow (it connects to the remote cluster and builds informers). This creates a wide race window.

When two secret-update events for the same remote cluster are processed in quick succession:

1. Both events create goroutines G1 and G2 that each call clusterUpdated for every registered handler.
2. G1 enters the slow constructor; G2 starts before G1 writes and reads old = clusters[X] = compA — the same stale value as G1 will read.
3. G1 finishes its constructor: writes compB, calls compA.Close() → close(compA.stop) ✓
4. G2 finishes its constructor: writes compC, calls compA.Close() → close(compA.stop) again → panic

The key issue is that the lock is released before the constructor begins (intentionally, to avoid holding it during slow I/O), but the read of old is placed after the constructor with no lock at all — creating a TOCTOU window.

Affected Versions

release-1.29. The release-1.29 branch did not receive the clusterUpdated rewrite from #58567 (which moved master to a pendingSwap-based approach that defers Close() until the new component syncs). PR #59876 (fixing #59875) was backported to release-1.29 and added a clusterClosing() call before the new goroutine starts, but that does not fix this panic: clusterUpdated on that branch still unconditionally calls old.Close() after the constructor, so a component closed by clusterClosing() is immediately double-closed when clusterUpdated runs.

Steps to Reproduce

Trigger two back-to-back kubeconfig secret updates for the same remote cluster within a short interval (e.g. automated credential rotation that produces two writes faster than the clusterUpdated constructor completes). The constructor duration is bounded by remote cluster connection latency, so the window can be seconds wide.