AF_PACKET raw ARP send fails with ENOBUFS on VLAN netdev over 802.3ad bond when interface is unnumbered or IPv4LL-addressed

Summary

We are running Talos Linux nodes with an 802.3ad bond (bond0) and VLAN subinterfaces. MetalLB in L2 mode was failing to send gratuitous ARP announcements on bond0.14 with:

sendto: no buffer space available

We reproduced the same failure outside of MetalLB using a privileged debug pod and Scapy, so this does not appear to be a MetalLB-specific issue.

The surprising behavior is:

• Raw ARP send through bond0.14 fails with ENOBUFS when the VLAN interface has no IPv4 address.
• Raw ARP send through bond0.14 also fails when the VLAN interface has an IPv4 link-local /32, such as 169.254.14.8/32.
• Raw ARP send through the same VLAN netdev succeeds once the VLAN interface has a private, non-link-local /32, such as 10.255.14.8/32.
• Sending the same VLAN 14 ARP frame as an explicitly tagged Dot1Q(vlan=14) frame through parent bond0 succeeds.

This suggests the physical NICs, LACP bond, switch trunk, and VLAN 14 fabric are working. The failure seems isolated to the Linux/Talos VLAN netdev egress path when the VLAN interface is unnumbered or only IPv4LL-addressed.

Environment

• Talos Linux
• Kernel/bonding driver reported as: Ethernet Channel Bonding Driver: v6.18.9-talos
• Interface layout:
  • bond0: 802.3ad bond over four physical NICs
  • bond0.12: VLAN 12, example public subnet 198.51.100.64/28
  • bond0.14: VLAN 14, example public subnet 203.0.113.240/28
  • bond0.101: management/internal VLAN

Bond configuration:

machine:
  network:
    hostname: node-08
    interfaces:
      - interface: bond0
        addresses:
          - 192.0.2.27/24
        routes:
          - network: 0.0.0.0/0
            gateway: 192.0.2.1
        bond:
          interfaces:
            - enp1s0f0
            - enp1s0f1
            - enp1s0f2
            - enp1s0f3
          mode: 802.3ad
          xmitHashPolicy: layer3+4
          lacpRate: fast
          miimon: 100
          updelay: 200
          downdelay: 200
        vlans:
          - vlanId: 12
            addresses:
              - 198.51.100.74/28
            routes: []
            dhcp: false
          - vlanId: 14
            addresses:
              - 169.254.14.8/32
            routes: []
            dhcp: false
          - vlanId: 101
            dhcp: true
            mtu: 1500
            routes: []
        dhcp: false

Original symptom

MetalLB speaker logs showed failures like:

{
  "caller": "announcer.go:221",
  "error": "writing \"OperationRequest\" gratuitous packet for \"203.0.113.252\": write packet 02:00:00:aa:bb:08: sendto: no buffer space available",
  "ip": "203.0.113.252",
  "level": "error",
  "msg": "failed to make gratuitous ARP announcement",
  "op": "gratuitousAnnounce"
}

The failing MAC 02:00:00:aa:bb:08 belongs to bond0 / bond0.14 on node-08.

Validation that the bond and VLAN trunk work

The LACP bond appears healthy:

• All four slaves are UP.
• All four slaves are in the same active aggregator.
• Actor/partner states show aggregating, in_sync, collecting, and distributing.
• No link failure counts.
• VLAN 14 ingress is visible on physical bond members.

We could see upstream ARP requests on VLAN 14 arriving at the node, for example:

Request who-has 203.0.113.252 tell 203.0.113.241
Request who-has 203.0.113.253 tell 203.0.113.241

So VLAN 14 is present on the trunk and ingress works.

Reproduction outside MetalLB

Using a privileged debug pod on the node, Scapy reproduced the same failure.

Failing case: raw ARP through bond0.14

from scapy.all import Ether, ARP, sendp, get_if_hwaddr

iface = "bond0.14"
vip = "203.0.113.252"
mac = get_if_hwaddr(iface)

pkt = Ether(dst="ff:ff:ff:ff:ff:ff", src=mac) / ARP(
    op=2,
    hwsrc=mac,
    psrc=vip,
    hwdst="ff:ff:ff:ff:ff:ff",
    pdst=vip,
)

sendp(pkt, iface=iface, count=1, verbose=True)

Result:

OSError: [Errno 105] No buffer space available

This matches the MetalLB failure.

Working case: explicitly tagged VLAN 14 frame through parent bond0

from scapy.all import Ether, Dot1Q, ARP, sendp, get_if_hwaddr

iface = "bond0"
vip = "203.0.113.252"
mac = get_if_hwaddr(iface)

pkt = Ether(dst="ff:ff:ff:ff:ff:ff", src=mac) / Dot1Q(vlan=14) / ARP(
    op=2,
    hwsrc=mac,
    psrc=vip,
    hwdst="ff:ff:ff:ff:ff:ff",
    pdst=vip,
)

sendp(pkt, iface=iface, count=1, verbose=True)

Result:

Sent 1 packets.

This suggests the NIC, bond, and switch fabric can carry VLAN 14. The failure is specific to sending through the VLAN netdev bond0.14.

Additional test matrix

We created temporary VLAN subinterfaces and tested raw ARP egress.

Temporary VLAN with no address

ip link add link bond0 name bond0.114 type vlan id 114
ip link set bond0.114 up

Scapy raw ARP through bond0.114 failed:

OSError: [Errno 105] No buffer space available

Temporary VLAN with IPv4 link-local /32

ip addr add 169.254.114.8/32 dev bond0.114

Scapy raw ARP through bond0.114 still failed:

OSError: [Errno 105] No buffer space available

Temporary VLAN with private non-link-local /32

ip addr add 10.255.114.8/32 dev bond0.114

Scapy raw ARP through bond0.114 succeeded:

Sent 1 packets.

Real VLAN 14 with private non-link-local /32

Replacing:

169.254.14.8/32

with:

10.255.14.8/32

on bond0.14 made the MetalLB-style Scapy ARP send succeed:

Sent 1 packets.

Things ruled out

We tested or inspected the following and they did not appear to be the root cause:

• MetalLB itself
• VLAN 14 missing from the switch trunk
• LACP bond failure
• Physical NIC egress failure
• TC filters on bond0.14
• qdisc drops on bond0.14
• VLAN TX offload on the physical slaves
• reorder_hdr
• Bond xmit_hash_policy

Specific tests:

• reorder_hdr off on bond0.14 did not fix the issue.
• Temporarily switching bond xmit_hash_policy from layer3+4 to layer2 did not fix the issue.
• Disabling tx-vlan-offload on all physical slaves did not fix the issue.
• No TC ingress/egress filters were present on bond0.14 or bond0.12.

Current hypothesis

The failure appears to be related to the Linux/Talos VLAN netdev egress path over an 802.3ad bond when the VLAN interface is either:

1. unnumbered, or
2. addressed only with IPv4 link-local 169.254.x.x.

Raw L2 ARP egress through the VLAN netdev returns ENOBUFS in those cases.

Assigning any private, non-link-local IPv4 /32 to the VLAN interface appears to make raw ARP egress work.

Workaround

We changed VLAN 14 from IPv4 link-local addressing to private dummy /32 addressing:

- vlanId: 14
  addresses:
    - 10.255.14.8/32
  routes: []
  dhcp: false

Suggested per-node pattern:

node-06: 10.255.14.6/32
node-07: 10.255.14.7/32
node-08: 10.255.14.8/32

This does not consume public addresses from the VLAN 14 public subnet, but it allows MetalLB to send gratuitous ARP on bond0.14.

Question for Talos/Sidero

Is this expected behavior?

More specifically:

• Should AF_PACKET raw egress through a VLAN netdev over an 802.3ad bond require a non-link-local IPv4 address on the VLAN interface?
• Is there known special handling for unnumbered or IPv4LL-addressed VLAN interfaces that could cause sendto() / raw packet sends to return ENOBUFS?
• Should Talos/networkd be configuring something differently for VLAN interfaces with 169.254.x.x/32 addresses?
• Is this more likely a Talos network configuration issue, a kernel regression, or expected Linux behavior?

The workaround is acceptable for now, but the behavior is surprising because raw L2 sends should not appear to depend on whether the VLAN netdev has a non-link-local IPv4 address.