Felix QoS controls: `bwcali*` IFB devices leak when a pod is deleted while Felix is not running

Author: pkossyfasCreated Sep 18, 2026Updated Sep 20, 2026

Expected Behavior

When a pod has a bandwidth annotation (qos.projectcalico.org/egressBandwidth or kubernetes.io/egress-bandwidth), Felix creates an IFB device bwcali<hash> in the host network namespace for the egress limit. We expect Felix to remove this device when the pod is deleted even if felix is unavailable (e.g. due to a calico-node pod rollout/restart) at that event. We also expect Felix to remove old bwcali* devices at start-up when they do not belong to a local workload anymore.

Current Behavior

Felix removes the IFB device only in maybeUpdateQoSBandwidth(old, nil) (felix/dataplane/linux/qos_controls.go). This function runs when Felix processes the WorkloadEndpoint removal.

If the pod is deleted while Felix is down (for example during a calico-node restart), the following happens:

The CNI DEL removes the veth together with the pod network namespace.
Nothing removes the bwcali* device.
When Felix starts again, the pod does not exist in the datastore anymore, so Felix does not get a removal event for it.
Felix has no start-up or periodic scan for bwcali* devices (we checked qos.go, qos_controls.go and endpoint_mgr.go on v3.30.4, v3.31.x, v3.32.0 and master). The cleanup added in #10159 works per veth and needs the veth to exist.

The device stays on the node until the node is recreated.

We see this in production. On one node with 29 days of uptime we found 1,558 orphaned bwcali* devices and only 3–6 running pods. The node runs short-lived Job pods with bandwidth annotations (about 6 pod deletions per minute), and calico-node was restarted every 10–15 minutes by the GKE vertical autoscaler. We monitored the node for two windows with no calico-node restart (about 190 pod deletions) and saw 0 leaked devices. In a 4-hour window we found 24 leaked devices, and 20 of them were created less than 100 seconds before a calico-node restart.

Side effects:

Every orphaned device is one more host interface with a TBF qdisc. With about 1,500 devices, kubelet /stats/summary grows to 2.2 MB and /metrics/cadvisor to 69 MB. Also every agent on the node that collects per-interface data (Elastic Agent, GKE gke-metrics-agent) needs more memory for each device.
The IFB name is bwcali + sha512(veth name)[:9], and the veth name comes from <namespace>.<pod name>. A new pod with the same name gets the same veth name, so Felix reuses the orphaned device and its old TBF state. We saw this in production for a device that was orphaned 3 days earlier.

Possible Solution

At start-up (and optionally at a regular interval), list the links with the bwcali prefix and delete the ones that do not match GetIfbDeviceName() of a local workload interface, or that are not the target of a mirred redirect filter on an existing cali* interface.
Or delete the IFB device when the workload interface disappears. Felix already gets interface state updates, so it can remove a bwcali* device when its veth is gone.
In addition, add a FelixConfiguration option to disable the QoS controls, or at least to ignore the kubernetes.io/*-bandwidth annotations. Today handleQoSControlsAnnotations() and the endpoint manager apply QoS with no option to turn it off (the only check is isQoSBandwidthSupported() for BPF). On clusters where the CNI chain still contains the upstream bandwidth plugin (for example GKE managed Calico, which ships v3.30.4 with a v3.29-style conflist), both implementations run for every annotated pod. They race on the veth (plugin type="bandwidth" failed (add): create qdisc: file exists, see #11666) and the pod ends up with two IFB devices and two TBF qdiscs.

Steps to Reproduce

We reproduced the leak on Calico v3.30.4 with both kubernetes.io/egress-bandwidth and qos.projectcalico.org/egressBandwidth.

On a cluster with Calico 3.30 or later (iptables dataplane), create a pod with an egress limit on node :

bash

kubectl run ifb-repro --image=busybox:1.36 --annotations=qos.projectcalico.org/egressBandwidth=30M --overrides='{"spec":{"nodeName":"<nodeName>"}}' -- sleep 3600

When the pod is Running, check on node <nodeName> that the IFB device exists. For pod default/ifb-repro the veth is cali0027f0645a2 (cali + sha1(default.ifb-repro)[:11]) and the IFB device is bwcali6514ff3bb (bwcali + sha512(cali0027f0645a2)[:9]):
bash
```
ip link show bwcali6514ff3bb
tc filter show dev cali0027f0645a2 parent ffff:
```
Delete the calico-node pod and the test pod with one command, so that the pod is deleted while Felix is down. On an idle node Felix is back in sync about 4 seconds after the calico-node pod is deleted, so the two deletions must run together:
bash
```
kubectl -n kube-system delete pod -l k8s-app=calico-node --field-selector spec.nodeName=<nodeName> --wait=false && kubectl delete pod ifb-repro --grace-period=0 --force
```

When the new calico-node pod is Ready, check node <nodeName> again. The veth is gone, but ip link show bwcali6514ff3bb still prints the device with its TBF qdisc. The Felix log shows Datastore in sync without ifb-repro, and it never shows Deleting QoS bandwidth state if present for this pod. The device is never removed.

Felix log of the new calico-node pod from our run (v3.30.4, pod annotated with qos.projectcalico.org/egressBandwidth=30M, calico-node pod deleted at 20:24:17 UTC together with the test pod). Felix sees the two interfaces at start-up, syncs with the datastore without the pod, and then only sees the veth go away:

2026-09-18 20:24:21.032 [INFO][57] felix/daemon.go 132: Felix starting up GOMAXPROCS=4 ... version="v3.30.4-gke.3"
2026-09-18 20:24:21.466 [INFO][57] felix/int_dataplane.go 1593: Linux interface state changed. ifIndex=141 ifaceName="cali0027f0645a2" state="up"
2026-09-18 20:24:21.467 [INFO][57] felix/int_dataplane.go 1593: Linux interface state changed. ifIndex=142 ifaceName="bwcali6514ff3bb" state="up"
2026-09-18 20:24:21.509 [INFO][57] felix/int_dataplane.go 2214: Datastore in sync, flushing the dataplane for the first time... timeSinceStart=477.447994ms
2026-09-18 20:24:21.509 [INFO][57] felix/endpoint_mgr.go 455: Workload interface came up, marking for reconfiguration. ifaceName="cali0027f0645a2"
2026-09-18 20:24:21.975 [INFO][57] felix/table.go 779: Found unexpected chain, marking for cleanup chainName="cali-tw-cali0027f0645a2" ipVersion=0x4 table="filter"
2026-09-18 20:24:21.975 [INFO][57] felix/table.go 779: Found unexpected chain, marking for cleanup chainName="cali-fw-cali0027f0645a2" ipVersion=0x4 table="filter"
2026-09-18 20:24:22.720 [INFO][57] felix/int_dataplane.go 1593: Linux interface state changed. ifIndex=141 ifaceName="cali0027f0645a2" state="down"
2026-09-18 20:24:22.723 [INFO][57] felix/int_dataplane.go 1593: Linux interface state changed. ifIndex=141 ifaceName="cali0027f0645a2" state=""

There is no WorkloadEndpointUpdate, no Local endpoint deleted and no Deleting QoS bandwidth state if present for default/ifb-repro in this log. Felix treats the veth as an unknown interface (Found unexpected chain) and never touches the IFB device.

State on the node after the new calico-node pod is Ready:

$ ip -o link show bwcali6514ff3bb
142: bwcali6514ff3bb: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1460 qdisc tbf state UNKNOWN ...
$ tc qdisc show dev bwcali6514ff3bb
qdisc tbf 1: root refcnt 2 rate 30Mbit burst 512Mb lat 453s
$ ip link show cali0027f0645a2
Device "cali0027f0645a2" does not exist.

Context

We run clusters where short-lived Job pods with bandwidth annotations are created and deleted continuously. calico-node was restarted every 10–15 minutes by the GKE vertical autoscaler. The leaked IFB devices grew to thousands per node, and the node monitoring agents were OOM-killed in a loop. It took us some time to trace the OOM kills back to the Felix QoS controls. A start-up cleanup in Felix would stop the accumulation, independent of how often calico-node restarts. An option to disable the QoS controls would let us keep one bandwidth implementation while the CNI chain still contains the kubernetes bandwidth plugin.

Your Environment

Calico version: v3.30.4 (GKE build calico/node:v3.30.4-gke.3). The relevant code is the same in v3.31.x, v3.32.0 and master.
Calico dataplane: iptables
Orchestrator version: Kubernetes v1.34.9 (GKE Standard), containerd 2.1.5
Operating System and version: Ubuntu 24.04.4 LTS, kernel 6.8.0-1055-gke

Source: projectcalico/calico

View original on GitHub View discussion on GitHub