dynamiccontroller exits or hangs when an edge watches a resource cloudcore can't watch
What happened:
When an edge node watches a resource through metaserver, cloudcore's dynamiccontroller creates an informer for that resource on first use. If creating it fails, cloudcore either exits or gets stuck.
ProcessApplication / processWatchApp → HandlerCenter.AddListener → ForResource → NewCommonResourceEventHandler:
informerPair, err := genericinformers.GetInformersManager().GetInformerPair(gvr)
if err != nil {
klog.Exitf("get informer for %s err: %v", gvr.String(), err)
}There are two failure modes:
The resource doesn't exist.
informers.forResourcecallsmapper.KindFor(gvr). For a CRD that isn't installed this returns an error, andklog.Exitfstops cloudcore. The edge agent keeps watch applications for 5 minutes after they close and re-sends them throughwatchapp/syncwhen it reconnects, so the same watch can make cloudcore crash again after every restart.The resource exists, but cloudcore's ClusterRole can't list/watch it.
addInformerPaircallscache.WaitForCacheSync(ifs.stopCh, ...), andstopChonly closes when cloudcore shuts down. The reflector keeps gettingforbidden, so the cache never syncs and the call never returns. During that wait it holds:informers.lock, which blocks every otherGetInformerPair/GetListercallerhandlerCenter.handlerLock, which blocks every otherForResource/DeleteListener
Center.Processruns synchronously on the singlereceiveMessagegoroutine. That means metaserver list, get and watch requests from every edge node stall, not just the node that sent the watch.
The failed informer entry also stays in informersByGVR, because it is stored before the sync wait. Any later caller gets an unsynced informer back with a nil error.
What you expected to happen:
A watch for a resource cloudcore can't inform should be rejected and the error returned to the edge, like any other failed Application. It should not stop cloudcore or block message handling for other nodes.
How to reproduce it (as minimally and precisely as possible):
Authorization is off by default (cloudHub.authorization.enable: false), so checkNodePermission does not reject the watch first.
Crash:
- Run cloudcore and an edge node with metaserver enabled, using the default configuration.
- From a pod on the edge node (or on the node itself), watch a CRD that is not installed in the cluster through metaserver:
kubectl --server=http://127.0.0.1:10550 get foos.example.com -w - cloudcore exits with
get informer for example.com/v1, Resource=foos err: no matches for .... - After cloudcore restarts and the edge reconnects, the watch is re-sent and cloudcore exits again.
Hang:
- Install a CRD that cloudcore's ClusterRole does not grant
list/watchon. - Watch it from the edge through metaserver as above.
- cloudcore stays up but logs
failed to list ... is forbiddenrepeatedly. Metaserver list/get/watch requests from all edge nodes stop getting responses.
Anything else we need to know?:
Suggested fix:
- Have
NewCommonResourceEventHandlerandForResourcereturn an error instead of callingklog.Exitf, and reject the Application so the edge gets the error. - Put a time limit on the cache-sync wait in
addInformerPair, and don't holdinformers.lockorhandlerLockwhile waiting. - If the sync fails, remove the informer entry so a later watch can retry cleanly.
Related, but not duplicates: #6632 and #5035 replace Exitf with panic, which still takes cloudcore down.
Environment:
Kubernetes version (use
kubectl version): anyKubeEdge version(e.g.
cloudcore --versionandedgecore --version): master (68e45c5ef1ee49f9ca3b61f93e0bbe915f398b11)- Cloud nodes Environment:
- Hardware configuration (e.g.
lscpu): N/A - OS (e.g.
cat /etc/os-release): N/A - Kernel (e.g.
uname -a): N/A - Go version (e.g.
go version): go1.23.12 - Others: default cloudcore config (authorization disabled)
- Hardware configuration (e.g.
- Edge nodes Environment:
- edgecore version (e.g.
edgecore --version): master - Hardware configuration (e.g.
lscpu): N/A - OS (e.g.
cat /etc/os-release): N/A - Kernel (e.g.
uname -a): N/A - Go version (e.g.
go version): go1.23.12 - Others: metaserver enabled
- edgecore version (e.g.
Source: kubeedge/kubeedge