Local cache restore copies outputs one file at a time: 61 s for 77,642 files on a Windows laptop whose disk is 6% busy

Current Behavior

Restoring outputs from the local cache copies one file at a time, so a large restore is bound by per-file latency while the disk sits idle. On a Windows laptop with a corporate security stack the copy alone takes a minute for a build whose outputs are 77,642 files.

Machine, 965-task restore, 77,642 files, 1.69 GB	Copy	Remove stale outputs and copy	Disk busy during the copy
Windows 11 laptop, 24 threads, NTFS	60.6 s	75.0 s	5.7%
M4 Mac, 16 cores, APFS	6.97 s	9.04 s	low

Three interleaved rounds each; the same tree copied by robocopy at 8 threads on the same laptop took 18.6 s, so the machine can do the work three times faster than Nx asks it to.

Why: NxCache.copyFilesFromCache is a synchronous napi method, so the Promise.all in finalizeCacheHits runs the restores one after another, and inside it copy_dir_all (packages/nx/src/native/cache/file_ops.rs) walks the tree on one thread with one fs::copy per file. Stale outputs are removed the same way, one remove_dir_all per output before its copy. Every file open passes through the filesystem filter drivers on the machine, and that cost is paid serially: during the Windows run the copying process used 0.87 cores and the antivirus service 1.40 while the other 22 threads idled.

Expected Behavior

A restore copies the entries of each directory in parallel, so the cost of the filter drivers overlaps instead of serialising, and the disk is the bound. With the copy parallelised on a pool a third of the CPU count the same restore measured 13.9 s on the laptop (4.4x) and 3.20 s on the Mac (2.2x); remove and copy 29.2 s and 4.43 s.

GitHub Repo

No response

Steps to Reproduce

1. A workspace whose cached outputs total tens of thousands of files (965 tasks, 77,642 files here). Any machine with a filesystem filter driver on the write path, such as an antivirus, shows it most.
2. Warm the local cache with one build. Delete the outputs (rm -rf dist).
3. Run the build again with NX_DAEMON=false so every task restores from the local cache, and time it; or time copyFilesFromCache directly with NX_VERBOSE_LOGGING=true.
4. Compare with copying the same tree with a multi-threaded copier (robocopy /MT:8 on Windows, cp -R per subdirectory in parallel on macOS).

Nx Report

Node : 26.8.1 OS : darwin-arm64 (also reproduced on win32-x64) Native Target : aarch64-macos yarn : 4.18.0

nx : 23.3.0-canary.20260915-071d621 (also 23.3.0-beta.0 and master at 8caae4d722) @nx/js : 23.3.0-canary.20260915-071d621 @nx/eslint : 23.3.0-canary.20260915-071d621 @nx/workspace : 23.3.0-canary.20260915-071d621 @nx/angular : 23.3.0-canary.20260915-071d621 @nx/jest : 23.3.0-canary.20260915-071d621 @nx/devkit : 23.3.0-canary.20260915-071d621 @nx/esbuild : 23.3.0-canary.20260915-071d621 @nx/eslint-plugin : 23.3.0-canary.20260915-071d621 @nx/node : 23.3.0-canary.20260915-071d621 @nx/playwright : 23.3.0-canary.20260915-071d621 @nx/storybook : 23.3.0-canary.20260915-071d621 @nx/vite : 23.3.0-canary.20260915-071d621 @nx/vitest : 23.3.0-canary.20260915-071d621 @nx/web : 23.3.0-canary.20260915-071d621 @nx/webpack : 23.3.0-canary.20260915-071d621

Failure Logs

No failure; the restore completes and the tree is correct. The finding is the time: 60.6 s to copy 1.69 GB on a machine that copies the same files in 18.6 s.

Package Manager Version

yarn 4.18.0

Operating System

Windows, macOS

Additional Information

#37085 parallelises the copy and the one-level removal, with the measurements above and the integrity checks in its body. #35172 parallelised the cache lookups; the copy was untouched.