Gateway API services no longer L2 announced in 1.20.2

Is there an existing issue for this?

• I have searched the existing issues

Version

equal or higher than v1.20.1 and lower than v1.21.0

What happened?

I upgraded from 1.20.1 to 1.20.2 and both of my gateway LoadBalancers stopped answering ARP. This took me a while to track down because there were no problems reported anywhere, even though all of my services were unreachable.

The gateways are Programmed, the routes are Accepted, the certs are Ready, LB IPAM still shows the IP assigned, the agents are Running with no errors, the l2-announcer's own health says job-l2-announcer-run: OK Running, and hitting the VIP with curl inside the cluster returned 200.

The problem finally presented itself after checking the leases:

$ kubectl get lease -A | grep l2announce
(nothing)

All three of my nodes report nothing via cilium-dbg:

$ cilium-dbg shell -- db/show l2-announce
IP   NetworkInterface

My LoadBalancers are externalTrafficPolicy: Local, so this seems to be due to #46399 which was backported as #48550. It was the only L2 change in 1.20.2. That PR makes lease allocation depend on where a service's backends are. Since Gateway API LoadBalancers are Envoy-backed and only have placeholder endpoints, no node ever has a local backend and the lease never gets allocated:

$ kubectl get endpointslices -n load-balancer
NAME                        ADDRESSTYPE   PORTS   ENDPOINTS         AGE
cilium-gateway-private-lb   IPv4          9999    192.192.192.192   198d
cilium-gateway-public-lb    IPv4          9999    192.192.192.192   198d

$ cilium-dbg shell -- db/show backends | grep -c cilium-gateway
0

How can we reproduce the issue?

1. Install Cilium 1.20.2 with l2announcements.enabled=true and kubeProxyReplacement=true.

2. Create a CiliumGatewayClassConfig that puts the generated Service on externalTrafficPolicy: Local, and a GatewayClass that uses it:

   yaml复制

   apiVersion: cilium.io/v2alpha1
   kind: CiliumGatewayClassConfig
   metadata:
     name: vlan
     namespace: load-balancer
   spec:
     service:
       type: LoadBalancer
       externalTrafficPolicy: Local

3. Create a Gateway served by that GatewayClass and let LB IPAM assign it an IP.

4. Add a policy that selects it. I use a serviceSelector in my own setup, but omitting it selects everything and is enough to reproduce:

   yaml复制

   apiVersion: cilium.io/v2alpha1
   kind: CiliumL2AnnouncementPolicy
   metadata:
     name: l2
   spec:
     loadBalancerIPs: true

5. Check for a lease:

   复制

   $ kubectl get lease -A | grep l2announce

On 1.20.2 there is never a lease and the VIP never answers ARP. On 1.20.1 with the same config the lease shows up and the VIP responds. Nothing is logged either way.

Cilium Version

1.20.2

Kernel Version

6.18.44-talos (Talos v1.13.9)

Kubernetes Version

v1.36.4

Regression

Yes. Worked in 1.20.1, breaks in 1.20.2.

Sysdump

I reverted back to 1.20.1 as soon as the cause was identified, so I didn't take a sysdump. The details above should cover tracking down the issue, but if you need me to run other commands or generate a sysdump, I can schedule some time to go back to 1.20.2 and generate one.

Relevant log output

No logs indicated any issue whatsoever. All log output and all statuses were green, but the gateway services were inaccessible.

Anything else?

Things I verified before submitting the issue:

• RBAC is set up properly, ServiceAccount is able to create, get, update, list, delete leases. auth can-i confirms it can list ciliuml2announcementpolicies.
• My Services have the correct selector label so the serviceSelector can find them.
• ciliuml2announcementpolicies.cilium.io CRD is still v2alpha1-only so there's no skew.
• enable-l2-announcements=true on both the agent and operator.
• kube-proxy-replacement=true is set.
• All three of my nodes are control plane nodes with exclude-from-external-load-balancers, but announcements still work on 1.20.1 with those same labels.
• My nodes' VLAN interface shows selected=true in db/show devices.

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct