#14556·k3s

Deploy controller modTime gate short-circuits checksum, missing manifest updates on immutable/epoch-mtime filesystems (NixOS /nix/store)

Author: ColaHikariCreated Aug 28, 2026Updated Sep 9, 2026

Environmental Info:

k3s v1.35.6+k3s1 (and confirmed still present in v1.36.4+k3s1)
OS: NixOS 26.11 (Linux, single-server k3s)

Describe the bug:

The deploy controller (pkg/deploy/controller.go) decides whether to re-apply a manifest using a modTime gate in listFilesIn():

go
modTime := files[path].ModTime()
if !force && modTime.Equal(w.modTime[path]) {
    continue   // skip deploy entirely
}

deploy() then has a second, content-based gate (checksum):

go
checksum := checksum(content)
if compareChecksum && checksum == addon.Spec.Checksum {
    return
}

The problem: the modTime gate short-circuits the checksum gate. On filesystems where file mtimes are not meaningful — e.g. NixOS /nix/store (all store files have a fixed 1970-01-01 mtime due to reproducible-build guarantees), and similar immutable/CI-output stores — the mtime never changes even when the file content changes. After nixos-rebuild, tmpfiles recreate the symlink to a new store path whose content is different, but its mtime is still 1970-01-01. modTime.Equal(w.modTime[path]) is always true, deploy() is never reached, and the manifest update is silently ignored.

Impact: any immutable / fixed-mtime manifest source never gets re-applied after content changes. Observed effects:

  • Adding an object to a manifest → never applied
  • Removing an object from a manifest → never pruned (wrangler prune is reliable when deploy() actually runs, verified by controlled test; it just never runs)

Workarounds that do NOT work for this case:

  • touch-ing the manifest (the issue #3711 workaround) — /nix/store is read-only
  • The only reliable workaround is restarting k3s (first loop runs with force=true), which is disruptive on production nodes

Steps to reproduce:

  1. On NixOS, link /var/lib/rancher/k3s/server/manifests/*.yaml to read-only /nix/store files (via tmpfiles)
  2. Start k3s, confirm initial apply
  3. Modify a manifest's content, nixos-rebuild switch (tmpfiles re-points the link to the new store path)
  4. Observe: k3s logs show no "Applying manifest" event; cluster resources unchanged (additions not created, removals not pruned)

Expected behavior: When modTime is unchanged but the file's content checksum differs from the Addon's recorded checksum, deploy() should still run (or the modTime gate should be skipped/softened when mtime is fixed/epoch). A reasonable fix: if modTime.Equal(w.modTime[path]), fall through to the checksum comparison instead of continue — the checksum is the authoritative content signal and is already persisted on the Addon CRD.

Additional context:

  • Related: #3711 "Auto-deploy Manifests Should Periodically Re-evaluate" (closed 2021, only fixed checksum persistence in #3920; the modTime gate remains)
  • Confirmed the modTime gate is still present in v1.36.4+k3s1