Bug: Goroutine leak and spurious reconnects in EdgeHub on connection drop
What happened:
In EdgeHub, the routeToCloud and keepalive goroutines can permanently leak upon reconnects. Furthermore, because reconnectChan is unbuffered, delayed errors from these leaked goroutines can cause spurious reconnects by sending a signal that tears down a newly established, healthy connection.
The old routeToCloud goroutine never exits because it is blocked on beehiveContext.Receive(). When it finally receives a message, it uses the shared eh.chClient (which has been updated to the new connection) and successfully sends the message, continuing its infinite loop. After N reconnects, there are N+1 routeToCloud goroutines leaked and competing to read from the channel.
Additionally, if the old goroutine encounters a send error before the new client is ready, it sends to the unbuffered eh.reconnectChan. This blocks the goroutine and will later trigger a spurious disconnect for the next healthy connection.
What you expected to happen:
Previous routeToCloud and keepalive goroutines should exit when the connection drops, ensuring only one active instance of each goroutine per active connection.
How to reproduce it (as minimally and precisely as possible):
- Start KubeEdge (
edgecoreandcloudcore). - Disconnect the network between edge and cloud (e.g., stop
cloudcoreor block port). - The
routeToEdgegoroutine detects the connection failure, sends toeh.reconnectChan, and exits. EdgeHub.Start()loop handles the reconnect, callsUnInit(), and sleeps.- Meanwhile, the
routeToCloudgoroutine is blocked onbeehiveContext.Receive(modules.EdgeHubModuleName)and does not exit. - Once the network is restored,
Start()creates a new connection, assigns it toeh.chClient, and spawns a NEW set ofrouteToCloud,routeToEdge, andkeepalivegoroutines. - Repeat the disconnect/reconnect process multiple times.
Anything else we need to know?: Root cause analysis:
EdgeHub.Start()spawnsrouteToCloudon every reconnect but provides no cancellation context to stop them.routeToCloudonly exits ifsendToCloudreturns an error. However, if it is blocked onbeehiveContext.Receive()during the disconnect, it will not hit the error path.eh.chClientis shared and updated on reconnect, so when the leakedrouteToCloudeventually receives a message, it uses the new connection, succeeds, and stays alive.eh.reconnectChan <- struct{}{}blocks if no receiver is ready. A delayed error will satisfy the<-eh.reconnectChanread of a subsequent healthy connection, causing a spurious reconnect cycle.
Possible fix direction:
- Pass a
context.ContexttorouteToCloud,routeToEdge, andkeepalivethat gets cancelled whenreconnectChanis triggered or whenUnInitis called. - Make
reconnectChanbuffered, or use a non-blocking send (select { case eh.reconnectChan <- struct{}{}: default: }) to prevent blocking leaked goroutines. - Ensure old goroutines exit by checking the context instead of relying solely on network write errors.
Environment:
Kubernetes version (use
kubectl version): N/AKubeEdge version(e.g.
cloudcore --versionandedgecore --version): Latest master branch- Cloud nodes Environment:
- Hardware configuration (e.g.
lscpu): Any - OS (e.g.
cat /etc/os-release): Any - Kernel (e.g.
uname -a): Any - Go version (e.g.
go version): Go 1.22+ - Others:
- Hardware configuration (e.g.
- Edge nodes Environment:
- edgecore version (e.g.
edgecore --version): Latest master branch - Hardware configuration (e.g.
lscpu): Any - OS (e.g.
cat /etc/os-release): Any - Kernel (e.g.
uname -a): Any - Go version (e.g.
go version): Go 1.22+ - Others:
- edgecore version (e.g.
Source: kubeedge/kubeedge