#7303·kubeedge

dynamiccontroller exits or hangs when an edge watches a resource cloudcore can't watch

Author: adity1rautCreated Sep 15, 2026Updated Sep 15, 2026
Labelskind/bug

What happened:

When an edge node watches a resource through metaserver, cloudcore's dynamiccontroller creates an informer for that resource on first use. If creating it fails, cloudcore either exits or gets stuck.

ProcessApplication / processWatchAppHandlerCenter.AddListenerForResourceNewCommonResourceEventHandler:

https://github.com/kubeedge/kubeedge/blob/68e45c5ef1ee49f9ca3b61f93e0bbe915f398b11/cloud/pkg/dynamiccontroller/application/eventhandler.go#L119-L123

go
informerPair, err := genericinformers.GetInformersManager().GetInformerPair(gvr)
if err != nil {
    klog.Exitf("get informer for %s err: %v", gvr.String(), err)
}

There are two failure modes:

  1. The resource doesn't exist. informers.forResource calls mapper.KindFor(gvr). For a CRD that isn't installed this returns an error, and klog.Exitf stops cloudcore. The edge agent keeps watch applications for 5 minutes after they close and re-sends them through watchapp/sync when it reconnects, so the same watch can make cloudcore crash again after every restart.

  2. The resource exists, but cloudcore's ClusterRole can't list/watch it. addInformerPair calls cache.WaitForCacheSync(ifs.stopCh, ...), and stopCh only closes when cloudcore shuts down. The reflector keeps getting forbidden, so the cache never syncs and the call never returns. During that wait it holds:

    • informers.lock, which blocks every other GetInformerPair / GetLister caller
    • handlerCenter.handlerLock, which blocks every other ForResource / DeleteListener

    Center.Process runs synchronously on the single receiveMessage goroutine. That means metaserver list, get and watch requests from every edge node stall, not just the node that sent the watch.

    https://github.com/kubeedge/kubeedge/blob/68e45c5ef1ee49f9ca3b61f93e0bbe915f398b11/cloud/pkg/common/informers/informer_manager.go#L218-L252

The failed informer entry also stays in informersByGVR, because it is stored before the sync wait. Any later caller gets an unsynced informer back with a nil error.

What you expected to happen:

A watch for a resource cloudcore can't inform should be rejected and the error returned to the edge, like any other failed Application. It should not stop cloudcore or block message handling for other nodes.

How to reproduce it (as minimally and precisely as possible):

Authorization is off by default (cloudHub.authorization.enable: false), so checkNodePermission does not reject the watch first.

Crash:

  1. Run cloudcore and an edge node with metaserver enabled, using the default configuration.
  2. From a pod on the edge node (or on the node itself), watch a CRD that is not installed in the cluster through metaserver: kubectl --server=http://127.0.0.1:10550 get foos.example.com -w
  3. cloudcore exits with get informer for example.com/v1, Resource=foos err: no matches for ....
  4. After cloudcore restarts and the edge reconnects, the watch is re-sent and cloudcore exits again.

Hang:

  1. Install a CRD that cloudcore's ClusterRole does not grant list/watch on.
  2. Watch it from the edge through metaserver as above.
  3. cloudcore stays up but logs failed to list ... is forbidden repeatedly. Metaserver list/get/watch requests from all edge nodes stop getting responses.

Anything else we need to know?:

Suggested fix:

  • Have NewCommonResourceEventHandler and ForResource return an error instead of calling klog.Exitf, and reject the Application so the edge gets the error.
  • Put a time limit on the cache-sync wait in addInformerPair, and don't hold informers.lock or handlerLock while waiting.
  • If the sync fails, remove the informer entry so a later watch can retry cleanly.

Related, but not duplicates: #6632 and #5035 replace Exitf with panic, which still takes cloudcore down.

Environment:

  • Kubernetes version (use kubectl version): any

  • KubeEdge version(e.g. cloudcore --version and edgecore --version): master (68e45c5ef1ee49f9ca3b61f93e0bbe915f398b11)

  • Cloud nodes Environment:
    • Hardware configuration (e.g. lscpu): N/A
    • OS (e.g. cat /etc/os-release): N/A
    • Kernel (e.g. uname -a): N/A
    • Go version (e.g. go version): go1.23.12
    • Others: default cloudcore config (authorization disabled)
  • Edge nodes Environment:
    • edgecore version (e.g. edgecore --version): master
    • Hardware configuration (e.g. lscpu): N/A
    • OS (e.g. cat /etc/os-release): N/A
    • Kernel (e.g. uname -a): N/A
    • Go version (e.g. go version): go1.23.12
    • Others: metaserver enabled