calico-node fails when ip_tables modules are not loaded

Environment:

• kubeadm kubernetes 1.36.3 (bare metal, single node)
• tigera-operator 3.32.1 (helm install)
• OS Fedora 44 (fully updated)
• Installation linuxDataplane: Nftables (may not be relevant, but the one used for testing)

Issue: calico-node fails to reach "Ready" state, generating 5MB/s logs filled with errors similar to

[WARNING][38914] felix/table.go 839: iptables-legacy-save command failed error=exit status 1 ipVersion=0x4 stderr="" table="filter"
[WARNING][38914] felix/table.go 904: iptables save failed error=exit status 1

Problem appears to be related to Fedora's "unified /usr/bin and /usr/sbin" (ref: https://fedoraproject.org/wiki/Changes/Unify_bin_and_sbin), and a recent update that has changed the system modprobe kernel path.

Attempting to run iptables-legacy-save --table=filter inside the failing calico-node container fails with the error:

iptables-save v1.8.10 (legacy): Cannot initialize: iptables who? (do you need to insmod?)

Tracing the iptables-legacy-save executable exposes the failing logic:

20847 openat(AT_FDCWD, "/proc/sys/kernel/modprobe", O_RDONLY|O_CLOEXEC) = 4
20847 getrandom("\x35\xda\x0e\xa9\x74\xa9\x08\x2b", 8, GRND_NONBLOCK) = 8
20847 brk(NULL)                         = 0x5594146a5000
20847 brk(0x5594146c6000)               = 0x5594146c6000
20847 read(4, "/usr/bin/modprobe\n", 4096) = 18
...
20848 execve("/usr/bin/modprobe", ["/usr/bin/modprobe", "ip_tables"], NULL) = -1 ENOENT (No such file or directory)
20848 exit_group(127)                   = ?

Clearly the iptables-legacy-save is reading the location of /usr/bin/modprobe from /proc/sys/kernel/modprobe mounted from the OS, and then using that path within the container, where modprobe is installed in /usr/sbin (not /usr/bin).

A recent update to Fedora 44 has apparently changed the location of modprobe exposed in /proc/sys/kernel/modprobe -- which works in Fedora 44 due to the unified /usr/bin, but fails in the container.

Running modprobe ip_tables (either in the OS or the container) loads the module and allows iptables-legacy-save to complete without error. Adding ip_tables (and ip6_tables) to /etc/module-load.d/iptables.conf provides a passable workaround.

Since the container executable relying on paths from the OS is clearly not reliable, the calico-node executable should probably directly pre-load the ip_tables/ip6_tables modules (if missing) before running iptables-legacy-save. Of course, the logic of calling the executable at all when those modules aren't loaded is debatable as there would be no iptables to examine, hence the executable could simply be skipped.

Either way, calico-node with fail on newer Fedora 44 (and probably soon other distributions with the unified /usr directories) unless some other system service isn't loading the legacy ip_tables modules, or the admin pre-loads the modules.