Confirmed: kubelet crashes on podman ZFS-graph-driver nodes — Kubernetes vendors cadvisor which lacks the zfs→VFS fallback (fixed on master, unreleased)

What happened

Running kind under podman (KIND_EXPERIMENTAL_PROVIDER=podman) on a host whose podman uses the native ZFS graph driver (storage.driver = "zfs"), the node comes up but kubelet crash-loops and the cluster never becomes ready:

container_manager_linux.go "Unable to get rootfs data from cAdvisor interface"
  err="cannot find filesystem info for device \"tank/podman/kind-probe/graph\""
kubelet.go "Failed to start ContainerManager"
  err="failed to get rootfs info: cannot find filesystem info for device ..."

Root cause (traced)

Inside the node, /dev/zfs is present but the zfs CLI is not installed in the node image. cadvisor's zfs plugin (fs/zfs/plugin.go GetStats) shells out to zfs list, which fails with exec ENOENT. In cadvisor v0.56.2 — the version vendored by the kubelet in kindest/node:v1.36.1 — that error path is:

capacity, free, avail, err := getZfsStats(device)
if err != nil {
    return nil, err     // no fallback
}

so cadvisor produces no filesystem info for the ZFS rootfs and kubelet exits 1.

Where cadvisor is vendored (correction)

To be precise about the fix path: kind/kindest/node does not vendor cadvisor — Kubernetes does. The kubelet in a kindest/node image is built from a Kubernetes release, and kubernetes/kubernetes go.mod pins the cadvisor module. So the fix must land in Kubernetes (via a cadvisor bump), after which kind build node-image picks it up.

• k8s v1.36.1 (the line behind kindest/node:v1.36.1) pins github.com/google/cadvisor v0.56.2 (released 2026-01-16).
• k8s master currently pins github.com/google/cadvisor/lib v0.60.5 (released 2026-07-11). (The module was split into a /lib submodule after the v0.56.x line.)

The fix (upstream cadvisor, master, unreleased)

cadvisor master fixed this in commit 0e107a38f5941b558c03ee3572906632a8198ae8 ("fs/zfs: fall back to VFS when zfs stats are unavailable", committed 2026-07-20) — GetStats now returns fs.ErrFallbackToVFS, which routes to statfs and succeeds:

capacity, free, avail, err := getZfsStats(device)
if err != nil {
    return nil, fs.ErrFallbackToVFS   // routes to VFS statfs; capacity reported; no crash
}

That commit is not in any cadvisor release yet: git tag --contains 0e107a38f5941b558c03ee3572906632a8198ae8 is empty — it landed 2026-07-20, after the latest tag v0.60.5 (2026-07-11). So neither the v0.56.2 pinned by k8s v1.36.1 nor the v0.60.5 pinned by k8s master contains it, and no released kubelet has the fallback.

Reproduction

Host: rootful podman with the native ZFS graph driver.

storage.conf:

[storage]
driver = "zfs"
graphroot = "/tank/podman/kind-probe/graph"
[storage.options.zfs]
fsname = "tank/podman/kind-probe/graph"

KIND_EXPERIMENTAL_PROVIDER=podman \
CONTAINERS_STORAGE_CONF=/path/to/storage.conf \
kind create cluster --name kind-probe --retain

Inside the node:

$ ls -l /dev/zfs
crw------- 1 root root 10, 249 /dev/zfs      # present
$ command -v zfs
                                             # ABSENT (exit 127) -> GetZfsStats fails
$ findmnt /
/  tank/podman/kind-probe/graph/<sha>  zfs

Full evidence writeup (node output, v0.56.2-vs-master cadvisor diff, and a passing end-to-end test) is here: https://gist.github.com/cmosetick/cd722e7f5897348046778432c19095db

Environment

• kind: v0.32.0, KIND_EXPERIMENTAL_PROVIDER=podman
• node image: kindest/node:v1.36.1 (kubelet v1.36.1, built from k8s that vendors cadvisor v0.56.2)
• podman: 6.0.2 (rootful), storage driver = zfs
• host ZFS: zfs-2.2.2, kernel 6.8.0-136-generic

Fix direction (once available)

Bump the cadvisor pin in Kubernetes to a release containing cadvisor 0e107a38f5941b558c03ee3572906632a8198ae8 (the fs/zfs VFS fallback); a kindest/node image built from that Kubernetes line then inherits the fix. Until cadvisor cuts such a release (tracked upstream: google/cadvisor#3922 "cut a release containing 0e107a38"), a node-image workaround is to ensure the zfs userspace CLI is present in the node image so GetZfsStats succeeds instead of exec-failing.

Upstream cadvisor tracking issue: https://github.com/google/cadvisor/issues/3922