BUG: Random failure and "handshake timeout" spam

What version of nebula are you using? (nebula -version)

1.10.0

What operating system are you using?

Flatcar Container Linux

Describe the Bug

This has happened quite a few times recently and I believe it is a Nebula bug.

I have a Kubernetes cluster running on top of a Nebula network. Nodes would randomly fall out the network and become partitioned until I manually restart the Nebula process, when it resumes normal operation. This renders my cluster extremely unstable and is very frustrating.

Might be related: I have used Nebula for less critical tasks before, and observed the same issue, especially on my laptop running Arch Linux. When I put the laptop to S3 suspend and resume later, the chance of Nebula failing is very high. Even when I do not suspend my laptop, it randomly fails every now and then. My servers running Ubuntu 24.04 have the same issue, so I believe it is distro agnostic.

When this happens, the logs show Nebula stuck indefinitely trying to re-establish handshake with a specific node. When I restart Nebula manually, it works again instantaneously so I am sure there is no underlying network issue. The node it fails to handshake with (in this case 10.18.1.3) is also perfectly reachable after the restart.

Logs from affected hosts

The logs are taken from the most recent failure, on the node where the failure occurred. You can see a massive chunk of "Handshake timed out" messages towards the end of the logs, before I restarted the service with systemctl.

Note: 10.0.2.207 is a Cilium address and 10.10.12.105 is the node's public-facing IP address (the node is behind NAT and that is the main reason why Nebula is used).

nebula.log

Config files from affected hosts

firewall:
  inbound:
    - host: any
      port: any
      proto: icmp
    - group: sysadmin
      port: any
      proto: any
    - group: k8s_node
      port: any
      proto: any
  inbound_action: reject
  outbound:
    - host: any
      port: any
      proto: any
  outbound_action: reject
handshakes:
  retries: 30
  try_interval: 1500
lighthouse:
  am_lighthouse: false
  hosts:
    - 10.18.0.1
  interval: 10
listen:
  host: 0.0.0.0
  port: 0
pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/host.crt
  disconnect_invalid: true
  key: /etc/nebula/host.key
punchy:
  punch: true
  respond: true
relay:
  am_relay: false
  relays:
    - 10.18.0.1
static_host_map:
  10.18.0.1: [REDACTED]:4242
tun:
  dev: neb0
  mtu: 1300
  unsafe_routes:
    - install: true
      mtu: 1300
      route: 192.168.64.43/32
      via:
        - gateway: 10.18.1.1
        - gateway: 10.18.1.2
        - gateway: 10.18.1.3