Bug: CNI plugin hard-fails with ENOENT on node boot — no retry during calico-node startup window

The CNI plugin performs a single os.Stat() call on /var/lib/calico/nodename with no retry and no backoff. If the file is absent — which it always is for a few seconds after a fresh node boot — the plugin returns immediately with:

plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory

The race window exists because of the ordering of two init steps:

• install-cni (initContainer) runs first: writes the CNI binary (/opt/cni/bin/calico) and the CNI conflist (/etc/cni/net.d/10-calico.conflist) to the host filesystem, then exits. containerd watches the conflist directory — once the conflist appears, containerd considers the CNI available and begins accepting RunPodSandbox calls.

• calico-node (main container) starts after install-cni exits. Its startup routine (startup.go → startup.Run()) must: connect to the Calico API client, query the node resource from the datastore, fetch the Kubernetes node object, configure IP/subnet/AS settings, apply node resource updates to the datastore, configure IP pools, and set global defaults — only then does it call utils.WriteNodeConfig() at startup.go:210, which writes /var/lib/calico/nodename. This takes several seconds under normal conditions.

During this window, sandbox creation attempts arrive. System DaemonSet pods (konnectivity-agent, CSI node drivers) tolerate NetworkUnavailable and begin retrying RunPodSandbox every ~15 seconds. Each attempt runs the CNI binary, which hits the single os.Stat() at plugin.go:187–192, gets ENOENT, and hard-fails.

The downstream impact scales with the number of concurrent failures. Each failed RunPodSandbox triggers a deferred ShutdownSandbox cleanup goroutine inside containerd (sandbox_run.go:302) that blocks on podSandbox.Wait() (shim ttrpc). Any concurrent metadata write then blocks on bbolt’s process-global rwlock (bbolt/db.go:145), which serialises all write transactions with no per-call timeout (plugin.go:44 Timeout: 0). Goroutines accumulate. Under sufficient load, the containerd process is killed (goroutine pool exhausted) and the ttrpc event channel to kubelet is permanently severed. The node appears dead to Kubernetes and requires an OS-level reboot to recover. systemctl restart containerd does not fix this — confirmed by direct observation: a fresh containerd process started and began discarding events within 90 minutes without a reboot.

Expected Behavior

When a node boots and calico-node is still initialising, the Calico CNI plugin should wait for /var/lib/calico/nodename to appear before processing sandbox requests. The code comment at the check site says “don’t start until it exists” — this is the correct intended behaviour. The plugin should poll with a bounded timeout (e.g. 30 seconds) so that the transient absence of the nodename file during the calico-node startup window does not cause a hard failure.

nodename_file_optional: false must remain — it is the correct safeguard ensuring a CNI call never proceeds on the wrong node. The fix must wait, not skip.

Current Behavior

The CNI plugin performs a single os.Stat() call on /var/lib/calico/nodename with no retry and no backoff. If the file is absent — which it always is for few seconds after a fresh node boot — the plugin returns immediately with:

plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory

Possible Solution

Replace the single os.Stat() check in cni-plugin/pkg/plugin/plugin.go — in both cmdAdd (~line 191) and cmdDel (~line 636) — with a poll loop with bounded timeout. A 500ms poll interval up to 30 seconds is sufficient to bridge the observed 14–22 second gap while keeping the wait bounded.

The cni-plugin/pkg/wait/ package already contains fsnotify-based file watching with a poll fallback — this infrastructure should be reused rather than reimplementing a raw sleep loop.

// Current — single check, hard-fails on ENOENT immediately
if _, err := os.Stat(nodeNameFile); err != nil {
    return fmt.Errorf(s, err)
}

// Proposed fix — poll with backoff, bounded timeout
const pollInterval = 500 * time.Millisecond
const timeout = 30 * time.Second
deadline := time.Now().Add(timeout)
for {
    if _, err := os.Stat(nodeNameFile); err == nil {
        break // file exists, proceed
    } else if !os.IsNotExist(err) {
        return fmt.Errorf("unexpected error checking nodename file: %w", err)
    }
    if time.Now().After(deadline) {
        return fmt.Errorf("timed out waiting for nodename file %s: "+
            "check that calico/node is running and healthy", nodeNameFile)
    }
    time.Sleep(pollInterval)
}

Important constraint: nodename_file_optional: false must be preserved. The fix must wait for the file, not skip the check — the nodename guard prevents traffic mis-routing to the wrong node and must remain a hard requirement.

Steps to Reproduce (for bugs)

Step 1 — Cordon the node Prevent the scheduler from landing workloads during the test window.

kubectl cordon $NODE

Step 2 — Create a test pod pinned to the node

The pod will sit in the scheduler queue and attempt RunPodSandbox immediately on node boot — landing squarely in the Calico not-ready window.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: cni-gap-test
spec:
  nodeName: $NODE
  tolerations:
  - operator: Exists   # tolerate all taints including NetworkUnavailable
  restartPolicy: Never
  containers:
  - name: test
    image: registry.redhat.io/ubi9/ubi-minimal:latest
    command: ["sleep", "60"]
EOF

Step 3 — Set up log capture before rebooting

Terminal 1 — watch pod status kubectl get pod cni-gap-test -w

Terminal 2 — watch events for CNI errors kubectl get events --field-selector involvedObject.name=cni-gap-test -w

Step 4a — Reboot the node (reliable, full window) or remove the nodename file (no reboot needed, faster iteration)

The second simulates the race directly. Felix re-writes the file within seconds, but the window is enough to catch a sandbox attempt.

Then immediately on your local machine:

kubectl delete pod cni-gap-test --force --ignore-not-found
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: cni-gap-test
spec:
  nodeName: $NODE
  tolerations:
  - operator: Exists
  restartPolicy: Never
  containers:
  - name: test
    image: registry.redhat.io/ubi9/ubi-minimal:latest
    command: ["sleep", "60"]
EOF

Step 5 — Observe the failure Pod events (Terminal 2):

Warning  Failed  RunPodSandbox  plugin type="calico" failed (add):
    stat /var/lib/calico/nodename: no such file or directory

Context

This bug was the primary root cause of a production outage on IBM IKS where the high number of System DaemonSet pods (50+) were starting before calico-node. Also, there was an additional issue which added blocked goroutines on the CNI delete path (sandbox cleanup) on top of the failures on the add path — double-sided pressure that most clusters never see.

The race window is architectural and always present on every Calico node boot. Whether it produces a deadlock depends on how many RunPodSandbox attempts land inside the window — which is determined by how many DaemonSet pods tolerate NetworkUnavailable (system DaemonSets do) and how quickly they begin retrying. On a busy cluster with a large scheduler backlog, the failure is near-certain on every reboot.

The source-level evidence is unambiguous:

startup.go:210 — WriteNodeConfig() is called after all datastore operations plugin.go:187–192 — single os.Stat(), immediate return on error, no retry loop The comment at the check site reads: “don’t start until it exists”

Your Environment

Your Environment Calico version: v3.30.7 (production incident); root cause code path confirmed present in cni-plugin/pkg/plugin/plugin.go Calico dataplane: iptables Orchestrator version: Kubernetes v1.34.10 (IBM IKS managed) Operating System and version: Ubuntu 24.04 (UBUNTU_24_64) Link to your project: n/a (IBM IKS managed cluster — not publicly accessible)