Eval cache re-hashes the whole source directory on every invocation, ignoring builtins.path/lib.fileset filters

Warm devenv shell cost grows without bound in the number of files inside any directory that evaluation coerced into a Nix store path, even when nothing has changed — and even when a builtins.path filter or lib.fileset excluded almost all of those files from the store path. In a repository whose coerced tree holds 681,435 files, a fully-valid cache hit spends 77 s of an 88 s run re-reading files that have not been touched and, in most cases, were never copied into the store at all.

Reproduction

https://github.com/schickling-assistant/devenv-eval-cache-repros/tree/main/repro-1

./repro.sh

The whole configuration is one line: env.SRC_DIR = "${./src}"; — and Part 5 repeats it with builtins.path { filter = …; } and lib.fileset.toSource.

Expected

A cache hit costs O(tracked inputs) stat calls. A directory whose mtime is unchanged and whose recorded content hash is current is not re-read. What is tracked is what Nix actually copied.

Actual

Deterministic, machine-load independent (strace -e trace=openat, 500 files in src/, cache valid, nothing changed):

devenv shell -- true      1000 opens / 500 files = 2 full tree walks
devenv tasks run noop     1000 opens / 500 files = 2 full tree walks
devenv tasks list          500 opens / 500 files = 1 full tree walk
devenv build                 0 opens / 500 files = 0 full tree walks

The shared config files are re-opened and re-hashed too, once per cached attribute:

devenv.nix     openat=10  stat*=28   per ONE warm `devenv shell -- true`
devenv.yaml    openat=13  stat*=31
devenv.lock    openat=12  stat*=29

Marginal wall-clock cost, interleaved A/B over an identical tree on disk with only devenv.nix differing:

marginal 480 ms at 20,000 files (median 577 − 97 ms) ⇒ ≈ 24 µs per file per invocation, unbounded. Note the payload is 40 KB: the cost is per-file, not per-byte.

On a real repository whose coerced tree holds 681,435 files / 1.5 GB, a cache-hit run took 88,066 ms of which 77,282 ms (88 %) was input validation: Evaluating shell itself was 4 ms and Evaluating devenv.config.task.config was 1 ms. Two walks at ≈38.6 s each ⇒ ≈57 µs/file. At that scale the eval cache is a net cost — a verified hit measured 8.1 s against 8.0 s for --no-eval-cache on an earlier, smaller version of the same tree.

The documented workarounds do not work

Both nixpkgs idioms for copying a minimal source select 1 file out of 2001 — and devenv hashes all 2001, twice, on every invocation:

builtins-path  Nix copied 1 file(s) into …-only-wanted
               devenv opened 4000 of the 2000 EXCLUDED files per warm invocation
lib-fileset    Nix copied 1 file(s) into …-source
               devenv opened 4000 of the 2000 EXCLUDED files per warm invocation

At field scale: a builtins.path filter explicitly excluding .devenv, .direnv, .git, dist, node_modules, result and target still had 803,363 files registered; a lib.fileset selecting ~124 files still had 681,435 files hashed.

That last case is worth spelling out, because it turns a performance bug into something closer to a correctness bug. The filter in question explicitly excluded the tool's own state directory — the one directory in the tree whose contents change on every invocation. Because the exclusion was ignored, that volatile directory stayed part of the tracked input, its hash changed every run, and the shell attribute's cache entry could therefore never be hit. The user did exactly the right thing, the exclusion was silently discarded, and the result was not a slow cache but a structurally dead one.

Narrowing one coercion site is also not sufficient in practice: after narrowing one, the same tree was re-registered by a second, unrelated call site elsewhere in the configuration.

Three mechanisms

1. Recursive inputs are excluded from the mtime fast path.

// devenv-eval-cache/src/eval_inputs.rs:259-264
let needs_rehash = file.recursive;
if mtime_unchanged && !needs_rehash {
    return Ok(FileState::Unchanged);
}

so validation always calls compute_directory_content_hash (devenv-cache-core/src/file.rs:222), a single-threaded WalkDir that blake3-hashes every byte of every file — no ignore rules, no size cap, no parallelism. extra_watch_paths are forced recursive at ffi_cache.rs:174-179, which is why devenv.nix/.yaml/.lock are re-hashed as well.

I understand why this is there — #2899 introduced it to fix the real staleness bugs in #2886 / #2893 / #3029 (a lib.fileset.gitTracked/directory-fileset source that was never rebuilt on edit), and the mtime of a directory genuinely does not change when a nested file is edited. The correctness fix is right; the problem is that it has no bound.

2. The same input set is validated once per cached attribute. CachingEvalService::get_cached → validate_inputs (devenv-eval-cache/src/caching_eval.rs:126, :230) runs independently for each key. A minimal project has 6–7 keys — config.cachix.{enable,pull,push}, shell, bash:build, devenv.config.task.config:build, plus build — all sharing the same inputs, so the whole set is validated once per key per invocation. (The three config.cachix.* keys come from three separate eval_field calls at devenv/src/devenv/cachix.rs:83,89,90 for three fields of one attrset.)

3. The tracker records the pre-filter directory.

// devenv-eval-cache/src/ffi_cache.rs:147
EvalOp::TrackedPath { source } | EvalOp::CopiedSource { source, .. } => (source, true),

source is the directory the filter was applied to, not the store path Nix produced, so builtins.path { filter = …; } and lib.fileset.toSource are invisible to input tracking. This is the part that leaves users with no workaround.

Possible directions

• Give CachingConfig.excluded_paths a user-facing surface in devenv.yaml. It is currently hardcoded to one entry (devenv-nix-backend/src/backend.rs:368), so there is no way for a user to say "this directory is build output, do not hash it" — and because the same tree can be coerced from several places, per-call-site narrowing is not a practical substitute. This is the one direction users can reach today.
• Track what Nix actually copied (the resulting store path's content) rather than the pre-filter source directory. This fixes the filter case outright and is arguably the correct semantics: the cache should depend on the derivation input, not on files that were excluded from it.
• Validate the union of inputs once per invocation and share the verdict across keys, rather than once per key.
• Cheap pre-check before the full walk: compare the max mtime and the (entry count, total size) of the tree, recorded alongside the content hash; only walk when that changes. Catches nested edits without reading bytes.
• Parallelise compute_directory_content_hash (it is already inside spawn_blocking, but one recursive input is one serial task).

Happy to attempt a PR — exposing excluded_paths in devenv.yaml plus adding the state directory to the default is the smallest change and would give users an escape hatch immediately; the tracker/pre-check changes are design decisions we'd want you to pick.

Versions

• devenv: 2.2.2+b8030c5 (x86_64-linux); also verified present on main (30058c52) — devenv-cache-core/src/file.rs has had no commits since v2.2.2 and eval_inputs.rs:260 is unchanged
• nix: Determinate Nix 3.17.3 / 2.33.3
• OS: Linux 6.18.43 x86_64 (32-core; measured under load, load average reported with every row in the repro output)