Daemon re-copies unchanged cached outputs after a restore: late watcher events and rescans drop recorded output hashes
Current Behavior
After a cache restore with the daemon running, the next build copies outputs back from the cache even though nothing changed. Two causes in packages/nx/src/daemon/server/outputs-tracking.ts:
- A change event under a recorded output is judged by its arrival time: more than two seconds after the record it drops the hash, inside two seconds it is ignored. During a large restore the watcher delivers the restore's own create and delete events well after the record, so the daemon forgets hashes the client has just stored. On a fast machine the other half bites: a real write within two seconds of the record is ignored, which is the race behind the intermittent e2e failure of
cache › should support using globs as outputs. - When the watcher reports dropped events (a rescan) the daemon clears every recorded hash. On Windows a large restore overflows the per-directory
ReadDirectoryChangesWbuffer, so the rescan lands in the middle of the restore.
| 965-task restore (74,570 files, 1.7 GB), second build started at once | Outputs re-copied |
|---|---|
| M4 Mac, three pairs | 10, 0, 32 of 965 |
| M4 Mac, second build after 60 s, three pairs | 26, 22, 22 of 965 |
| Windows 11 laptop, three pairs | 4, 4, 5 of 965 |
| Windows 11 laptop, one observed run with a rescan during the restore | 531 of 965 |
The unpatched daemon also re-copies about 5 of 965 tasks on every steady-state build for the first reason. Watcher lag measured up to 24 s after the record on the Mac and 50 s on the Windows laptop.
Expected Behavior
A build after a restore leaves every unchanged output in place, and a rescan does not cost a full re-copy of everything restored so far.
Steps to Reproduce
- Workspace with many cached tasks whose outputs total tens of thousands of files (965 tasks, 74,570 files here). Daemon on.
rm -rf dist(or the outputs), thennx build <root project>so every task restores from the local cache.- Run the same build again immediately. Count tasks reported as copied from the cache rather than "existing outputs match the cache, left as is". Repeat after 60 s: the largest outputs re-copy every time.
- For the rescan case, stop the daemon process during the restore (SIGSTOP on macOS) while writing a burst of files elsewhere so the watcher drops events, then continue it and build: on the unpatched daemon every hash recorded before the rescan is gone.
Nx Report
Node : 26.8.1 OS : darwin-arm64 (also reproduced on win32-x64) Native Target : aarch64-macos yarn : 4.18.0
nx : 23.3.0-canary.20260915-071d621 (also 23.3.0-beta.0 and master at 8caae4d722) @nx/js : 23.3.0-canary.20260915-071d621 @nx/eslint : 23.3.0-canary.20260915-071d621 @nx/workspace : 23.3.0-canary.20260915-071d621 @nx/angular : 23.3.0-canary.20260915-071d621 @nx/jest : 23.3.0-canary.20260915-071d621 @nx/devkit : 23.3.0-canary.20260915-071d621 @nx/esbuild : 23.3.0-canary.20260915-071d621 @nx/eslint-plugin : 23.3.0-canary.20260915-071d621 @nx/node : 23.3.0-canary.20260915-071d621 @nx/playwright : 23.3.0-canary.20260915-071d621 @nx/storybook : 23.3.0-canary.20260915-071d621 @nx/vite : 23.3.0-canary.20260915-071d621 @nx/vitest : 23.3.0-canary.20260915-071d621 @nx/web : 23.3.0-canary.20260915-071d621 @nx/webpack : 23.3.0-canary.20260915-071d621
Failure Logs
daemon.log shows the record batches, then [WATCHER]: Processing file changes in outputs batches arriving up to 24 s later; on a rescan, The outputs watcher reported dropped events; clearing recorded output hashes.
Package Manager Version
yarn 4.18.0
Operating System
macOS, Windows
Additional Information
Two PRs: #37087 dates every event by the path's mtime with no arrival-time window; #37088 verifies hashes against the files after a rescan instead of clearing them.
Source: nrwl/nx