LoadBalancer Service: stale backend in eBPF map ~20-34s after pod deletion, independent of L2-announcement lease

Is there an existing issue for this?

• I have searched the existing issues

Version

equal or higher than v1.19.7 and lower than v1.20.0

What happened?

After deleting a Pod backing a LoadBalancer Service (StatefulSet, 3 replicas, one pod per node via podAntiAffinity), new TCP connections to the Service's external IP consistently fail or hang for ~20-34 seconds, even though 2 of 3 backends remain healthy and Ready throughout, and the Kubernetes EndpointSlice for the Service is updated correctly and promptly.

This is not correlated with L2 Announcement lease ownership. We specifically tested killing the pod running on the node that currently holds the cilium-l2announce Lease (leaseDurationSeconds: 15), versus killing a pod on a different node (non-leader). Both scenarios produced equivalent downtime:

• Leader-node pod killed: 34s downtime, Lease holder unchanged throughout (same node, same AGE, no re-election occurred)
• Non-leader-node pod killed: 32s downtime
• Additional runs (mixed pods/nodes): 20-30s downtime, one outlier run recovered immediately

This rules out L2 Announcement leader re-election as the cause, and points instead to a delay in Cilium's own eBPF service/backend map reconciliation after pod deletion, independent of L2 announcement state.

Captured via cilium-dbg bpf lb list ~2s after kubectl delete pod (pod IP already deleted, replacement not yet Ready): the deleted pod's IP was still present as one of 3 active backend slots for the affected LoadBalancer service, alongside the 2 genuinely healthy backends.

We also confirmed via direct pod-IP connection (bypassing the LoadBalancer Service entirely) that the backend application itself (a 3-node clustered MQTT broker, VerneMQ) remained fully functional and correctly clustered throughout all tests. Connecting directly to a surviving pod's IP during the outage window succeeded immediately with 0s downtime. This isolates the issue purely to the Cilium LoadBalancer Service datapath, not the application or the Kubernetes control plane.

Configuration relevant to this issue:

• kubeProxyReplacement: true
• ipam.mode: cluster-pool, pod CIDR 10.42.0.0/16
• L2 Announcements enabled (l2announcements.enabled=true, externalIPs.enabled=true), no BGP control plane
• Masquerading: IPTables mode (not BPF masquerading)
• CiliumLoadBalancerIPPool with a /28 block
• Affected Service: type LoadBalancer, single TCP port 1883, backing a 3-replica StatefulSet spread one-pod-per-node via required podAntiAffinity on kubernetes.io/hostname
• sessionAffinity: None on the Service (confirmed, ruled out as a factor)
• Operator replicas: 2

How can we reproduce the issue?

1. Deploy a 3-replica StatefulSet (one pod per node via podAntiAffinity) behind a type LoadBalancer Service (single TCP port) on a kubeProxyReplacement + L2Announcement Cilium setup, no BGP.
2. Confirm via cilium-dbg service list that the LoadBalancer service shows all 3 pod backends as active.
3. From a client on a separate node, loop new TCP connections against the Service's external IP once per second (we used mosquitto_pub, a one-shot MQTT publish per invocation, genuinely new TCP connection each time, not a reused/reconnecting client).
4. In parallel, run kubectl delete pod on one of the three pods, repeat with different target pods on different nodes, including the node that currently holds the relevant cilium-l2announce Lease.
5. Observe: connection attempts fail (Connection Refused or hang) for approximately 20-34 seconds regardless of which pod/node was killed, including when the L2-announcement Lease holder's own pod was killed (Lease itself never changed holder).
6. Cross-check cilium-dbg bpf lb list against the service frontend IP during the failure window: the deleted pod's IP remains listed as an active backend slot for several seconds after the Kubernetes EndpointSlice has already dropped it.
7. As a control, connect directly to a surviving pod's IP (bypassing the Service) during the same outage window. This succeeds immediately, confirming the application/cluster itself is healthy and the delay is isolated to the Service's eBPF datapath.

Cilium Version

cilium-dbg version Client: 1.19.6 9a898243 2026-07-16T13:44:52+00:00 go version go1.25.12 linux/arm64 Daemon: 1.19.6 9a898243 2026-07-16T13:44:52+00:00 go version go1.25.12 linux/arm64

Kernel Version

Raspberry Pi 4, Ubuntu Server 26.04 LTS, ARM64 (aarch64)

Kubernetes Version

k3s v1.32.4+k3s1, 3 control-plane + 3 worker nodes, 6 nodes total

Regression

No response

Sysdump

cilium-sysdump-20260825-020923.zip

Relevant log output

cilium-dbg bpf lb list output, service ID 124 (192.168.1.171:1883/TCP, LoadBalancer), captured ~2s after kubectl delete pod vernemq-0 (pod IP was 10.42.0.165):

192.168.1.171:1883/TCP (0) 0.0.0.0:0 (124) (0) [LoadBalancer]
192.168.1.171:1883/TCP (2) 10.42.2.110:1883/TCP (124) (2)
192.168.1.171:1883/TCP (1) 10.42.1.34:1883/TCP (124) (1)
192.168.1.171:1883/TCP (3) 10.42.0.165:1883/TCP (124) (3)

Slot 3 still points at 10.42.0.165, the pod already deleted by kubectl at this point. kubectl get endpointslices for the same Service, checked at the same time, had already correctly dropped this endpoint and listed only the 2 genuinely healthy pods as ready: true.

Timing data across 6 reproduction runs (kubectl delete pod timestamp to first successful new TCP connection via the Service IP):

- Run 1: vernemq-2 killed, node pi-core-2, not lease holder at the time, downtime 29s
- Run 2: vernemq-0 killed, node pi-core-1, not lease holder, downtime 28s
- Run 3: vernemq-0 killed, node pi-core-1, not lease holder, downtime 23s
- Run 4: vernemq-0 killed, node pi-core-1, not lease holder, downtime 23s
- Run 5: vernemq-2 killed, node pi-core-2, lease holder: yes, downtime 34s
- Run 6: vernemq-0 killed, node pi-core-1, not lease holder, downtime 32s

Lease detail for run 5 (leader-node pod killed): the cilium-l2announce Lease's holderIdentity and AGE remained unchanged before and after the test, no re-election occurred, ruling out L2-announcement leader-election timing as the cause.

Anything else?

This may be related to or a regression variant of:

• https://github.com/cilium/cilium/issues/22782 (Cilium LoadBalancer stops working with no other active session)
• https://github.com/cilium/cilium/issues/37452 (One node, when restarting pods, loadbalancer service stops working)

Neither matches our setup exactly (we don't use BGP, and the issue self-heals after ~20-34s rather than being permanent), but the symptom class (stale backend in bpf lb list after Pod deletion, EndpointSlice already correct, L2-lease unaffected) looks similar and may share a root cause in service/backend map reconciliation timing.

Note: the Version dropdown had no option matching our exact version 1.19.6, jumping from v1.18.13 to v1.19.7. Selected the closer option; actual confirmed version is 1.19.6 per the Cilium Version field above.

Happy to provide additional captures (hubble observe, cilium-dbg bpf ct list, additional sysdumps, or repeat the leader/non-leader test with more samples) if useful for triage.

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct