BUG: One-way blackhole when peer is only reachable via relay (stale HostInfo.remote never cleared)

Summary

HostInfo.remote (the cached "best known direct underlay" for a peer) can become stale and pin a tunnel to a dead direct underlay address indefinitely. The tunnel is kept marked alive by inbound relay traffic, so the connection manager never tears it down or triggers a rehandshake. The result is a persistent one-way blackhole: the local node keeps sending to a dead direct address while the peer is reaching us fine via a relay.

Process restart (or any event that drops the HostInfo) recovers the tunnel because the new tunnel is established via relay only and HostInfo.remote stays unset, so sendNoMetrics falls into the relay branch.

Root cause (two interacting invariants)

1. sendNoMetrics in inside.go prefers direct over relay whenever hostinfo.remote.IsValid(). There is no fallback to relay if the direct path is dead:

   go复制

   useRelay := !remote.IsValid() && !hostinfo.remote.IsValid()
   ...
   if remote.IsValid() {
       err = f.writers[q].WriteTo(out, remote)
   } else if hostinfo.remote.IsValid() {
       err = f.writers[q].WriteTo(out, hostinfo.remote)   // pinned to dead underlay
   } else {
       // Try to send via a relay
       ...
   }

2. handleHostRoaming in outside.go only updates hostinfo.remote for non-relayed inbound packets:

   go复制

   func (f *Interface) handleHostRoaming(hostinfo *HostInfo, via ViaSender) {
       if !via.IsRelayed && hostinfo.remote != via.UdpAddr {
           ...
           hostinfo.SetRemote(via.UdpAddr)
       }
   }

   Relayed packets still call connectionManager.In(hostinfo), so the tunnel is considered alive purely from relay traffic.

3. connectionManager.makeTrafficDecision does not distinguish direct vs. relay traffic in its in/out tracking, so inTraffic == true from relay-only liveness is enough to keep the tunnel alive forever. tryRehandshake only fires for cert version drift, never for stale remote.

Two manifestations

Forward case: direct died, peer fell back to relay, we stayed pinned to direct

1. Tunnel is up direct. hostinfo.remote = X.
2. Underlay X becomes unreachable from the peer (NAT timeout, cellular reattach, route flap).
3. Peer keeps the tunnel alive via a relay it has access to.
4. Our node decrypts relayed packets → In(hostinfo) → tunnel is "alive". handleHostRoaming skips because via.IsRelayed == true, so hostinfo.remote stays at X.
5. Our outbound packets go direct to X and are blackholed.
6. No rehandshake fires (cert is fine), no teardown fires (relay traffic refreshes liveness).
7. Tunnel is permanently one-way broken.

Reverse case: roamed to a bad direct address, then direct broke

1. Tunnel was relay-only (or direct via X). hostinfo.remote either invalid or X.
2. A single direct packet arrives from address Y (transient NAT rebind, brief cell handoff, asymmetric path).
3. handleHostRoaming accepts and sets hostinfo.remote = Y. There is no validation that Y is bidirectionally reachable.
4. Y is dead for outbound (NAT entry already reaped, asymmetric path, etc). Peer keeps reaching us via relay.
5. Same stuck state as the forward case, but reached via a different history.

RoamingSuppressSeconds = 2 only suppresses roaming back to the previous remote. It does not prevent roaming to a bad address in the first place.

Minimum reproduction

Three nodes:

• LH — lighthouse.
• R — relay.am_relay: true, advertises itself as a relay.
• A and B — clients with use_relays: true and relay.relays: [R], both pointing at LH.

Forward case

1. Bring up all four nodes. From A ping B's overlay address; tunnel establishes direct (each side learns the other's underlay from the lighthouse). Confirm A → B works and A's host SSH command shows Remote: populated with B's direct underlay address.

2. Block the direct path between A and B only, leaving the relay path intact. For example on the host running A:

   bash复制

   sudo iptables -I OUTPUT 1 -d <B-direct-underlay-ip> -j DROP
   sudo iptables -I INPUT  1 -s <B-direct-underlay-ip> -j DROP

3. From B, generate a small steady stream of overlay traffic toward A (e.g. ping <A-overlay> once a second). Because A is not reachable directly anymore, the packets travel via R. A decrypts them, marks the tunnel alive.

4. Ping A → B over the overlay. Expected: packets fall back to the relay and reach B. Actual: all outbound packets from A go to the cached dead direct underlay; nothing reaches B.

5. A's tunnel to B stays "alive" indefinitely. connectionManager never decides deleteTunnel, no rehandshake fires.

6. Restart A (or otherwise drop the HostInfo) and the next handshake establishes via relay only; ping recovers.

Reverse case

Same topology. Establish the tunnel via relay only first (block direct between A and B before they handshake). Then:

1. Briefly unblock the direct path long enough for one direct encrypted packet from B to reach A (a single test packet is enough).
2. A's handleHostRoaming sets hostinfo.remote to B's direct underlay.
3. Re-block direct in both directions. Keep relay path up.
4. From A, ping B. Outbound goes to the just-pinned direct address and is blackholed. Tunnel stays alive on relay traffic. No recovery.

Why the existing safeguards don't catch this

• connectionManager.makeTrafficDecision only checks inTraffic and outTraffic aggregated across both paths.
• tryRehandshake is gated on cert/version state, not on remote health.
• pendingDeletion requires inTraffic == false, which never happens while relay traffic flows.
• Punchy punches hostinfo.remote (or all remotes), but that doesn't change which address sendNoMetrics uses for data.
• lighthouse.remote_allow_list can prevent acceptance of inbound roams, but it does not validate outbound deliverability of an already-accepted remote.

Suggested fix

A few possible directions:

1. Track direct vs. relay traffic separately on HostInfo (e.g. an inDirect atomic.Bool set only in the non-relay decrypt path). In connectionManager.makeTrafficDecision, if the host is the primary, has had outbound traffic, and has had no direct inbound for K ticks, clear hostinfo.remote (forcing the relay branch in sendNoMetrics) and trigger a rehandshake.
2. Validate new direct addresses before pinning. In handleHostRoaming, instead of immediately calling SetRemote(via.UdpAddr), send a test packet to the candidate and only pin once a reply is observed from that exact address. This eliminates the reverse manifestation (roam-to-bad).
3. Add an outbound deliverability heartbeat. When hostinfo.remote.IsValid() and we've sent N packets without any direct inbound reply, fall back to relay automatically until a direct path proves itself again.

(1) is the minimal change and addresses both manifestations. (2) is complementary and prevents the reverse case from ever entering the bad state.

Environment

• Reproduced on upstream nebula 02f78b099c726dc940fb65a9834158a16ec70562 (master, late September 2025). The relevant code paths in inside.go, outside.go, and connection_manager.go are unchanged on current main/release, so the issue should still apply.
• Linux clients, IPv6 overlay, mixed v4/v6 underlays, cipher: aes, use_relays: true, am_relay: false on the affected clients.

Workarounds

• Restart the affected client to clear HostInfo.remote.
• A liveness probe (e.g. periodic overlay ping to a canary peer) that restarts the process on failure recovers from this without manual intervention.
• Enabling stats: type: prometheus and alerting on per-peer outbound/inbound byte ratio surfaces stuck tunnels before they are reported by users.

Happy to follow up with logs or a failing e2e test if helpful.