BUG: HostQuery fails to resolve underlay addresses for secondary overlay IPs in dual-stack certs (`udpAddrs=[]`)

What version of nebula are you using? (nebula -version)

1.11.0

What operating system are you using?

Linux (Debian 13), amd64

Describe the Bug

(I run a self-hosted control plane server that speaks the dnclient API and uses the same binaries.)

Dual-stack deployments use separate per-address tunnel lookups by design (migration). The issue is that lighthouse resolution for a given overlay address does not populate that address's lookup, so traffic to valid overlay IPs fails with udpAddrs=[] even while the same peer is reachable on its sibling overlay address.

This fails randomly. The same host pair can work for hours (or days) and then lose reachability on one overlay address with no config change, no cert rotation, and no obvious tunnel teardown in logs. It affects many hosts across the fleet, not a single bad node or a reproducible-on-demand setup. A dnclient restart on either side usually restores connectivity, which makes it look transient, but the underlying lookup gap can come back unpredictably.

My desktop lost connection to a large number of IPv6 overlay addresses and a dnclient restart fixed it.

Symptom: Handshake timed out vpnAddrs=[<overlay-ip>] udpAddrs=[] means no Handshake message sent for that address. Remote host receives nothing. The sibling overlay address for the same peer often still works at the same moment.

Why this feels random:

• Tunnels frequently establish on IPv4 first; IPv6 monitoring/targets keep working until they don't, with no Tunnel status dead / Close tunnel / Dropping tunnel for that peer beforehand (unlike peers that fail via normal active probing in the same time window).
• A peer can be registered at the lighthouse with both overlay addresses at boot, reachable on IPv4/LAN, and still time out on IPv6 ~14 hours later with zero traffic on the failing path.
• The same initiator can ping the IPv6 overlay at 100% loss while pinging the IPv4 overlay of the same peer in the same minute gives full udpAddrs, relay, success.
• Restarting dnclient on either side typically fixes it by re-registering addresses and repopulating caches=but there is no reliable way to predict when the next drop will happen.

Querying lighthouse for overlay address X (whether IPv4 or IPv6) should return the underlay addresses and fill X's RemoteList, enabling handshake to that address. What actually happens is that the lookup for the secondary/IPv6 overlay address never gets underlay addresses. Handshake sends nothing. Sibling address (usually cert-primary IPv4) resolves fine via the same lighthouse.

Restarting dnclient on either side typically restores reachability by re-registering addresses and repopulating caches.

Logs from affected hosts

Client C (initiator) - metrics repro, same minute

Sep 01 20:54:35 dekstop dnclient[1240]: Handshake timed out vpnAddrs=[fdec:d8be:746d::2f] udpAddrs=[]
Sep 01 20:55:14 dekstop dnclient[1240]: Handshake timed out vpnAddrs=[fdec:d8be:746d::2f] udpAddrs=[]
Sep 01 20:56:24 dekstop dnclient[1240]: Handshake message sent vpnAddrs=[172.0.2.131] udpAddrs="[<public-ip>:15310 <public-ip>:29619 10.0.0.43:52420]"
Sep 01 20:56:24 dekstop dnclient[1240]: Attempt to relay through hosts vpnAddrs=[172.0.2.131] relays="[fdec:d8be:746d::2 fdec:d8be:746d::5]"
Sep 01 20:56:24 dekstop dnclient[1240]: Handshake message received vpnAddrs=[172.0.2.131] certName=metrics certVersion=1 from="<public-ip>:4242 (relayed)"
Sep 01 20:56:28 dekstop dnclient[1240]: Handshake timed out vpnAddrs=[fdec:d8be:746d::2f] udpAddrs=[]

No Handshake message sent lines for ::2f - underlay list never populated.

Peer D (metrics) - same minute, v4 only

Sep 01 20:56:24 metrics dnclient[175]: handleCreateRelayRequest relayFrom=172.0.1.1 relayTo=172.0.2.131
Sep 01 20:56:24 metrics dnclient[175]: send CreateRelayResponse relayFrom=172.0.1.1 relayTo=172.0.2.131

No lines for 746d::2f.

Host A (monitor) - original incident, boot

Sep 01 03:07:05 host-a dnclient[2391020]: Handshake timed out vpnAddrs=[172.0.2.3] udpAddrs="[10.0.0.5:57326 <public-ip>:16145 <public-ip>:31252]"
Sep 01 03:07:34 host-a dnclient[2391020]: Handshake message received vpnAddrs=[172.0.2.3] from=10.0.0.5:42597 certName=host-b certVersion=1
Sep 01 03:07:34 host-a dnclient[2391020]: Handshake message sent vpnAddrs=[172.0.2.3] from=10.0.0.5:42597 certName=host-b certVersion=1

Host A (monitor) - failure ~14h later

Sep 01 16:59:40 host-a dnclient[2391020]: Handshake timed out vpnAddrs=[fdec:d8be:746d::e] udpAddrs=[]

(No Tunnel status / Close tunnel / Dropping tunnel for host-b in 03:07–17:00.)

Host B (monitored peer) - failure window

(no dnclient lines for host-a or fdec:d8be:746d::e at failure time)

Lighthouse - host-b registration at boot

Sep 01 03:07:33 lighthouse dnclient[750]: Handshake message received vpnAddrs="[172.0.2.3 fdec:d8be:746d::e]" from=<public-ip>:37313 certName=host-b certVersion=2

Config files from affected hosts

• All hosts: dual-stack overlay (IPv4 + IPv6 networks on cert)
• Many peers still on cert v1 protocol during migration; lighthouses reachable over v1 tunnels
• Peers: dynamic listen port, not relays, LAN 10.0.0.0/20
• Monitoring/clients often target IPv6 overlay addresses; tunnels frequently establish on IPv4 first
• Lighthouse: standard lighthouse, no static_host_map for affected peers
• No remote_allow_list blocking LAN underlay