Difficult to not use iptables-legacy under rootless podman
What happened:
Creating a cluster using KIND_EXPERIMENTAL_PROVIDER=podman in rootless mode, and the kind-control-plane always starts linking the iptables binaries to iptables-legacy, even if the ip{6,}_tables modules aren't loaded.
The issue appears to be in /usr/local/bin/entrypoint, in the select_iptables() function. It checks for the count of rules in iptables-nft and iptables-legacy, with the most rules selecting the iptables version. If the count is the same, the default is legacy.
When started with docker, docker creates rules in iptables-nft, and so the select_iptables() function chooses "nft"
However, podman doesn't create any iptables rules, and the function selects "legacy" by default.
When kubelet starts, it uses the host iptables version to select which iptables version to create the KUBE-KUBELET-CANARY chain. In rootful mode, this loads the ip{6,}_tables modules if not already loaded, and creates the chain.
In rootless mode however, the modules can't be loaded (as the "privileged" container is still running as non-root). This means that the CANARY chain cannot be created, and other pods that depend on it can fail.
What you expected to happen:
When a cluster is started, and no chains are present, the selection of iptables should be more intelligent. For example, the presence of kernel modules might indicate which iptables versions are supported on the host.
How to reproduce it (as minimally and precisely as possible):
Setup a host (or VM) but don't load the ip_tables or ip6_tables kernel modules.
Setup rootless podman on a non-root user.
Run: KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster
After the cluster is started, run: podman exec kind-control-plane iptables --version
The version will be legacy, but the ip_tables modules will still not be loaded
run: podman exec kind-control-plane iptables-save
There will be no output, no KUBE-KUBELET-CANARY chain has been created.
Anything else we need to know?:
I managed to coerce the entrypoint to choose nft mode by creating an oci hook as follows:
{
"version": "1.0.0",
"hook": {
"path": "/bin/bash",
"args": ["bash", "-c", "iptables-nft -N TEST-CANARY; iptables-nft -A TEST-CANARY -s 127.0.0.55"]
},
"when": {
"commands": [ "^/usr/local/bin/entrypoint$" ]
},
"stages": ["startContainer"]
}The presence of the rule selected nft iptables and everything worked correctly (KUBE-KUBELET-CANARY chain was created).
The reason I discovered this was attempting to run calico failed, as it uses the same technique of counting iptables rules between nft and legacy (with legacy being the tie-breaker). On a functional system, this works since the presence of the KUBE-KUBELET-CANARY chain will correctly identify the iptables in use, but when kubelet can't create it, then chaos ensues :)
Again, a simple lsmod check could be added to the select_iptables() function to "break the tie" if both iptables versions have no rules, which hopefully would at least prevent an unusable iptables version from being selected.
Environment:
- kind version: 0.33.0
- Runtime info:
host:
arch: amd64
buildahVersion: 1.43.2
cgroupControllers:
- cpuset
- cpu
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.2.1-2.fc44.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.2.1, commit: '
cpuUtilization:
idlePercent: 98.17
systemPercent: 0.55
userPercent: 1.28
cpus: 56
databaseBackend: sqlite
distribution:
distribution: fedora
version: "44"
eventLogger: journald
freeLocks: 2046
hostname: cluster3
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 7.1.7-200.fc44.x86_64
linkmode: dynamic
logDriver: journald
memFree: 11646746624
memTotal: 67393966080
networkBackend: netavark
networkBackendInfo:
backend: netavark
defaultNetwork: podman
dns:
package: aardvark-dns-1.17.1-1.fc44.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.17.1
package: netavark-1.17.2-1.fc44.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.17.2
ociRuntime:
name: crun
package: crun-1.28-1.fc44.x86_64
path: /usr/bin/crun
version: |-
crun version 1.28
commit: 54f16ffbefcd022bf032af768b5c5ce075c18bfc
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +JSON_C
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20260728.gf8df3f1-2.fc44.x86_64
version: |
pasta 0^20260728.gf8df3f1-2.fc44.x86_64
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: true
path: /run/user/1000/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: ""
package: ""
version: ""
swapFree: 53111721984
swapTotal: 53666115584
uptime: 524h 48m 5.00s (Approximately 21.83 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
search:
- docker.io
store:
configFile: /home/devel/.config/containers/storage.conf
containerStore:
number: 1
paused: 0
running: 1
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/devel/.local/share/containers/storage
graphRootAllocated: 210241560576
graphRootUsed: 79895597056
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 42
runRoot: /run/user/1000/containers
transientStore: false
volumePath: /home/devel/.local/share/containers/storage/volumes
version:
APIVersion: 5.8.4
BuildOrigin: Fedora Project
Built: 1782432000
BuiltTime: Thu Jun 25 17:00:00 2026
GitCommit: 5431df23c742e5edea35bef34eed696f4db0106b
GoVersion: go1.26.4-X:nodwarf5
Os: linux
OsArch: linux/amd64
Version: 5.8.4- OS: Fedora 44
- Kubernetes version: Client Version: v1.36.3
Source: kubernetes-sigs/kind