1.35: Node reamins 'NotReady' after network disconnect

Summary

After a network disconnect a node remains 'NotReady' until microk8s is restarted on that node.

What Should Happen Instead?

Node returns to 'Ready' without having to restart micronk8s

Reproduction Steps

1. Fresh ubuntu 22.04.5 OS (firewall disabled)
2. Create a 3 node cluster (snap install microk8s, microk8s join...)
3. See that all 3 nodes are 'Ready' in 'kubectl get no'
4. Disconnect host3 by unplugging network cable (VMs used here so link disconnect)
5. Observe 'kubectl get no' until host3 marked as 'NotReady' ~1 min
6. Reconnect network cable on host3. Confirm host3 can ping host1/host2.
7. Observe that host3 never returns to 'Ready' state.
8. To bring host3 back to 'Ready' microk8s stop / microk8s start must be performed on the host3.

Introspection Report

Attached inspection reports from all 3 nodes.

I notice the following lines on host3 syslog that gets repeated that look relevant in updating node status in dqlite.

Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/masterleases/10.173.130.213 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node-nlg2q.1883d9e60b8807d3 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node.1883d9c5198e7ee5 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node-m4brh.1883d9df1d2a4748 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-kube-controllers.1883d9c4d7376729 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/default/dailybuild4-host1.1883d9c3c28b4e77 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node-7lmvd.1883d9d145cf05a5 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/default/dailybuild4-host1.1883d9c3c2d81bb4 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node-nlg2q.1883d9e58f35ea44 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/coredns-78894c95f4-q62rl.1883d9cb855c0107 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/default/dailybuild4-host1.1883d9c3c151e8bb for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/default/dailybuild4-host2.1883d9e4d9368fe6 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node.1883d9de9eba07b3 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-kube-controllers-676854dc5d-89fvm.1883d9c51e245adf for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node-7lmvd.1883d9cbba2e2adb for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="error in txn: update transaction failed for key /registry/leases/kube-node-lease/dailybuild4-host3: exec (try: 0): context canceled" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node-m4brh.1883d9dee2290173 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node-7lmvd.1883d9cb4f900780 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/coredns-78894c95f4-q62rl.1883d9d4e757dd42 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="error in txn: update transaction failed for key /registry/leases/kube-node-lease/dailybuild4-host3: exec (try: 0): context canceled" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="error in txn: update transaction failed for key /registry/masterleases/10.173.130.213: exec (try: 0): context canceled" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="failed to delete /registry/events/kube-system/calico-node-nlg2q.1883d9e59b2d4012 for TTL: exec (try: 0): context canceled, retrying" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=error msg="error in txn: update transaction failed for key /registry/leases/kube-system/apiserver-o5p3gggce3s67d2x5pdrmhvgey: exec (try: 0): context canceled" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-kubelite[16786]: W1223 13:15:13.214279 16786 lease.go:265] Resetting endpoints for master service "kubernetes" to [10.173.130.211 10.173.130.212 10.173.130.213] Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=warning msg="[go-dqlite] attempt 91: server 10.173.130.211:19001: dial: dial tcp 10.173.130.211:19001: operation was canceled" Dec 23 13:15:13 dailybuild4-host3 microk8s.daemon-k8s-dqlite[15035]: time="2025-12-23T13:15:13Z" level=warning msg="[go-dqlite] attempt 91: server 10.173.130.211:19001: context canceled"

Can you suggest a fix?

no

Are you interested in contributing with a fix?

no

inspection-report-20251223_132504-host3.tar.gz inspection-report-20251223_132535-host2.tar.gz inspection-report-20251223_132505-host1.tar.gz