Linux 1.102.2 regression: sustained direct traffic triggers BCM57414 firmware reset, DMAR faults, and host network outage

What is the issue?

After upgrading a Proxmox VE host from Tailscale 1.98.9 to 1.102.2, sustained traffic over a direct Tailscale path reproducibly triggers a fatal failure of a dual-port Broadcom BCM57414 (bnxt_en) adapter.

The failure sequence includes:

• Fatal firmware reset event from bnxt_en
• Intel IOMMU DMA Read NO_PASID faults from PCI functions 01:00.0 and 01:00.1
• NETDEV WATCHDOG and transmit-queue timeouts
• cascading HWRM request/ring cleanup failures
• loss of host network connectivity requiring a host reboot

This host and backup path had operated reliably for approximately one year. Downgrading only Tailscale to 1.98.9, restoring normal NIC offloads, and leaving the hardware, NIC firmware, kernel, network topology, and peer unchanged eliminated the failure:

This appears distinct from #19777:

• This host is running Proxmox kernels 7.0.14-4-pve and 7.0.14-11-pve, which should contain the relevant upstream UDP GSO fixes.
• The observed result is a fatal NIC firmware reset with IOMMU DMA faults and total network loss, rather than asymmetric throughput degradation.
• The affected adapter is a Broadcom BCM57414 using bnxt_en, not one of the NIC/driver combinations identified in #19777.
• The differentiating variable here is Tailscale 1.98.9 versus 1.102.2; the failure reproduced on both tested 7.0.14 Proxmox kernels.

The timing of the regression is nevertheless notable because Tailscale's kernel-7 UDP GSO handling changed in the v1.100+ timeframe. Please investigate whether the workaround or another packet/offload change between 1.98.9 and 1.102.2 can expose a BCM57414 firmware/driver defect.

I recognize that a legal userspace packet/offload pattern should not cause a NIC firmware crash or invalid DMA access, so this may involve both a Tailscale regression trigger and a Broadcom firmware/bnxt_en robustness issue.

Steps to reproduce

Warning: reproduction may cause a complete host network outage requiring reboot.

1. Install Tailscale 1.102.2 on the affected Proxmox host with normal NIC offload settings.

2. Confirm the remote peer uses a direct path with tailscale status or tailscale ping.

3. Start an iperf3 server on the remote Tailscale peer on TCP port 5001.

4. On the affected host, run:

   bash复制

   iperf3 -c <TAILSCALE_PEER_IP> -p 5001 -t 180

5. Monitor kernel messages for DMAR, bnxt_en, firmware reset, NETDEV WATCHDOG, and TX timeout.

The production workload that also reproduces the issue is a Proxmox VM backup to the same Proxmox Backup Server over Tailscale.

Expected behavior

The sustained transfer completes without affecting the physical adapter. Any NIC or driver error should recover without taking the host network offline.

Actual behavior

Traffic stops, the NIC reports a fatal firmware-reset event, the IOMMU reports an invalid DMA read, transmit queues time out, and bnxt_en begins returning HWRM failures. The bonded physical network becomes unusable until the host is rebooted.

Representative log sequence:

Aug 10 14:50:52 kernel: bnxt_en 0000:01:00.1 eno2np1: Fatal firmware reset event, data1: 0x201, data2: 0xda27, min wait 1300 ms, max wait 4200 ms
Aug 10 14:50:52 kernel: DMAR: [DMA Read NO_PASID] Request device [01:00.0] fault addr 0xfed2c000 [fault reason 0x06] PTE Read access is not set
Aug 10 14:50:57 kernel: bnxt_en 0000:01:00.0 eno1np0: NETDEV WATCHDOG: CPU: 6: transmit queue 6 timed out 5441 ms
Aug 10 14:50:57 kernel: bnxt_en 0000:01:00.0 eno1np0: TX timeout detected, starting reset task!
Aug 10 14:50:57 kernel: bnxt_en 0000:01:00.0 eno1np0: hwrm vnic set tpa failure rc for vnic 0: fffffff3
Aug 10 14:50:57 kernel: bnxt_en 0000:01:00.0 eno1np0: hwrm_ring_free type 1 failed. rc:fffffff0 err:3

An independent boot captured the same core signature:

Aug 10 14:43:07 kernel: DMAR: [DMA Read NO_PASID] Request device [01:00.0] fault addr 0xfc583000 [fault reason 0x06] PTE Read access is not set
Aug 10 14:43:07 kernel: bnxt_en 0000:01:00.1 eno2np1: Fatal firmware reset event, data1: 0x201, data2: 0xda27, min wait 1300 ms, max wait 4200 ms

The second PCI function also subsequently failed:

Aug 10 14:59:55 kernel: DMAR: [DMA Read NO_PASID] Request device [01:00.1] fault addr 0xfc9f0000 [fault reason 0x06] PTE Read access is not set
Aug 10 15:00:00 kernel: bnxt_en 0000:01:00.1 eno2np1: NETDEV WATCHDOG: CPU: 6: transmit queue 4 timed out 5384 ms
Aug 10 15:00:00 kernel: bnxt_en 0000:01:00.1 eno2np1: TX timeout detected, starting reset task!

Controls and eliminated variables

• Reproduced on both 7.0.14-4-pve and 7.0.14-11-pve.
• High-throughput non-Tailscale IPv4 and IPv6 tests at more than 4 Gb/s did not reproduce the failure.
• Disabling GRO, GSO, TX checksum offload, or TSO on both physical ports did not prevent the failure.
• Disabling tx-udp-segmentation on tailscale0 delayed but did not prevent the failure. This is a clue, not proof of root cause.
• Both physical links were healthy at 10 Gb/s, with zero bond link-failure counts, after rollback.
• The Tailscale path to the backup peer was direct UDP, not DERP-relayed.

Are there any recent changes that introduced the issue?

Yes. Tailscale was upgraded from 1.98.9 to 1.102.2 immediately before the failures began. Package history records:

Upgrade: tailscale:amd64 (1.98.9, 1.102.2)

Downgrading only Tailscale to 1.98.9 eliminated the failure under the same focused test and production backup workload.

OS

Linux

OS version

• Debian GNU/Linux 13 (trixie)
• Proxmox VE 9.2.0
• pve-manager 9.2.4
• Tested kernels: 7.0.14-4-pve and 7.0.14-11-pve

Tailscale version

• Failing: 1.102.2
• Known good: 1.98.9 (4fb758c39ae5b208b974af14ba6bc896a250394c)

Other software / hardware

• Dual-port Broadcom BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
• PCI ID 14e4:16d7, revision 01
• PCI functions 0000:01:00.0 and 0000:01:00.1
• Driver: bnxt_en
• NIC firmware: 220.0.59.0, package 220.0.83.0
• Both physical links negotiated at 10 Gb/s
• Physical ports are members of an 802.3ad bond using LACP fast and a layer3+4 transmit hash

Bug report

BUG-c989bce0d4d8024ecafb5158bd88684e8b987dc1fbf3a6a0278a8bf546a9dd3a-20260810203322Z-0f0c6117ac5d6645

This bug-report marker was generated with tailscale bugreport --diagnose after rollback, while the host was healthy on 1.98.9. It is not a bracketed recording of the destructive 1.102.2 reproduction.

I can provide the complete diagnostic collector privately. I have not attached it publicly because it contains hostnames, tailnet addresses, public endpoint addresses, interface configuration, and peer information. If a tailscale bugreport --record reproduction or instrumented build is required, I can coordinate it during an outage window because the failure may require a host reboot.

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

No response

OS version

No response

Tailscale version

No response

Other software

No response

Bug report

No response