Deploy controller modTime gate short-circuits checksum, missing manifest updates on immutable/epoch-mtime filesystems (NixOS /nix/store)
Environmental Info:
k3s v1.35.6+k3s1 (and confirmed still present in v1.36.4+k3s1)
OS: NixOS 26.11 (Linux, single-server k3s)Describe the bug:
The deploy controller (pkg/deploy/controller.go) decides whether to re-apply a manifest using a modTime gate in listFilesIn():
modTime := files[path].ModTime()
if !force && modTime.Equal(w.modTime[path]) {
continue // skip deploy entirely
}deploy() then has a second, content-based gate (checksum):
checksum := checksum(content)
if compareChecksum && checksum == addon.Spec.Checksum {
return
}The problem: the modTime gate short-circuits the checksum gate. On filesystems where file mtimes are not meaningful — e.g. NixOS /nix/store (all store files have a fixed 1970-01-01 mtime due to reproducible-build guarantees), and similar immutable/CI-output stores — the mtime never changes even when the file content changes. After nixos-rebuild, tmpfiles recreate the symlink to a new store path whose content is different, but its mtime is still 1970-01-01. modTime.Equal(w.modTime[path]) is always true, deploy() is never reached, and the manifest update is silently ignored.
Impact: any immutable / fixed-mtime manifest source never gets re-applied after content changes. Observed effects:
- Adding an object to a manifest → never applied
- Removing an object from a manifest → never pruned (wrangler prune is reliable when
deploy()actually runs, verified by controlled test; it just never runs)
Workarounds that do NOT work for this case:
touch-ing the manifest (the issue #3711 workaround) —/nix/storeis read-only- The only reliable workaround is restarting k3s (first loop runs with
force=true), which is disruptive on production nodes
Steps to reproduce:
- On NixOS, link
/var/lib/rancher/k3s/server/manifests/*.yamlto read-only/nix/storefiles (via tmpfiles) - Start k3s, confirm initial apply
- Modify a manifest's content,
nixos-rebuild switch(tmpfiles re-points the link to the new store path) - Observe: k3s logs show no "Applying manifest" event; cluster resources unchanged (additions not created, removals not pruned)
Expected behavior:
When modTime is unchanged but the file's content checksum differs from the Addon's recorded checksum, deploy() should still run (or the modTime gate should be skipped/softened when mtime is fixed/epoch). A reasonable fix: if modTime.Equal(w.modTime[path]), fall through to the checksum comparison instead of continue — the checksum is the authoritative content signal and is already persisted on the Addon CRD.
Additional context:
- Related: #3711 "Auto-deploy Manifests Should Periodically Re-evaluate" (closed 2021, only fixed checksum persistence in #3920; the modTime gate remains)
- Confirmed the modTime gate is still present in
v1.36.4+k3s1
Source: k3s-io/k3s