Streaming JSON dominates the JS side of compiles - add a buffered fast path
The --cpu-prof pass of the Hardhat 3 + EDR profiling campaign shows streaming JSON I/O dominating the JS side of every compile. @streamparser/json (BufferedString.appendChar/toString, tokenizer.write) plus json-stream-stringify are 60–72% of active JS time in compile runs across all 11 scenarios; in lidofinance-dual-governance's cold compile, three streamparser functions alone are 55% of active JS.
In wall-clock terms the leverage sits on smaller compiles. Cold compiles are only ~1–9% active JS, so streaming JSON is ~1–5% of their wall. Min-deps incremental compiles are ~35–50% active JS, of which streaming is ~33–46% — ~12–20% of wall.
Character-by-character JS parsing is typically 10–50× slower than native JSON.parse on a buffered string, and it drives the allocation and GC churn perf observed during compiles. The streaming design in hardhat-utils (fs.ts:417,487) exists for memory safety — solc output can exceed V8's ~512 MB string limit — so the constraint is real, but it is paid on every file regardless of size. Solidity-test runs already read artifacts through the buffered path; this is a compile, build-info and cache concern.
Direction
A size-thresholded fast path in hardhat-utils: stat first, and below a threshold (~100–256 MB, well under the V8 string limit) read the file whole and JSON.parse it, keeping the streaming path above it. On the write side, JSON.stringify plus a single write with a RangeError fallback to streaming. This preserves the public API and the memory-safety guarantee, and the streaming dependencies then load only on the rare big-file path — which also removes their module-load cost from startup.
Worth exploring as a near-term win before committing to a full build-system port to Rust.
Implementation plan
# Add a buffered JSON fast path to hardhat-utils fs
## Problem
The `--cpu-prof` pass of [Profiling Hardhat 3 e2e scenarios](https://app.notion.com/p/nomicfoundation/Runtime-Profiling-2026-08-05-3b3578cdeaf5808eafdff4e22ba425d0?source=copy_link) shows the JS side of every compile dominated by streaming JSON I/O. `@streamparser/json` (`BufferedString.appendChar`/`toString`, `tokenizer.write`) plus `json-stream-stringify` account for **60–72% of active JS time in compile runs** across all 11 scenarios; in lidofinance-dual-governance's cold compile, three streamparser functions alone are 55% of active JS.
In wall-clock terms the leverage sits on smaller compiles: cold compiles run only ~1–9% active JS, so streaming JSON is ~1–5% of their wall, while min-deps incremental compiles run ~35–50% active JS and streaming is ~33–46% of it — **~12–20% of wall**. Character-by-character JS parsing is typically 10–50× slower than native `JSON.parse` on a buffered string, and it drives the allocation and GC churn perf observed during compiles.
Absolute stakes per run are moderate — solc dominates compiles overall — but this is the single biggest fixable JS cost in the compile path, and it multiplies across every compile, cache read/write and artifact load. Solidity-test runs already read artifacts through the buffered `readBinaryFile`/`readJsonFile`/`parseJsonBytes` path; the streaming cost is confined to compile, build-info and cache call sites.
## Root cause
- `packages/hardhat-utils/src/fs.ts:417` `readJsonFileAsStream` and `:487` `writeJsonFileAsStream`, built on `@streamparser/json-node` and `json-stream-stringify` (lazily imported via `packages/hardhat-utils/src/internal/bytes.ts`).
- Call sites: `packages/hardhat/src/internal/builtin-plugins/solidity/build-system/{compiler/compiler.ts,cache.ts,build-system.ts}` and `packages/hardhat/src/internal/builtin-plugins/node/helpers.ts` — solc standard-JSON output, build-info files, artifacts, cache entries.
- The streaming design exists for memory safety: solc output for big projects can be hundreds of MB and V8 caps strings at ~512 MB (`ERR_STRING_TOO_LONG`). The fix must respect that constraint, not remove it.
## Task
1. Measure first: micro-benchmark `readJsonFileAsStream` against `JSON.parse(await readFile(path, 'utf8'))` (and the write-side equivalents) on real artifacts and build-info files from an initialised scenario clone, at 1 KB, 1 MB, 50 MB and the largest available.
2. Implement a **size-thresholded fast path** in `hardhat-utils`:
- `readJsonFileAsStream`: `stat` first; below a threshold (propose ~100–256 MB — measure where the buffered win ends, and stay well under the V8 string limit), read the file whole and `JSON.parse` it; above it, keep streaming.
- `writeJsonFileAsStream`: for objects that stringify under the threshold, `JSON.stringify` plus a single write, catching `RangeError`/invalid-string-length to fall back to streaming.
- Preserve the public API, the error conventions (`HardhatError`/`ensureError`), any tmp-file-plus-rename atomicity, and the lazy loading of the streaming dependencies — which should now load only on the rare big-file path, removing their module-load cost from startup.
3. Check behavioural differences between the streaming parser and `JSON.parse`: BigInt, revivers, duplicate keys, and on the write side `undefined`, Maps and key ordering versus `JSON.stringify`.
4. Extend the existing `packages/hardhat-utils` unit tests with threshold-boundary cases (just below, just above, malformed JSON on both paths, concurrent reads). Run `pnpm lint`, `pnpm build`, `pnpm test` in `packages/hardhat-utils` and `packages/hardhat`.
## Verification (before/after)
Profile before and after with `pnpm profiler` (a bare `pnpm profiler` prints its usage; see `scripts/README.md`):
```bash
pnpm build
pnpm profiler --scenario ./end-to-end/lidofinance-dual-governance \
--prepare "cold compile" \
--command "edit & compile Solidity test with min deps: test/unit/scripts/launch/TimeConstraints.t.sol" \
--mode js --init --use-local
```
Expect: streamparser and json-stream-stringify functions gone from (or reduced to noise in) the `.cpuprofile`, and incremental-compile `wallSeconds` measurably lower — cross-check with `hyperfine -w 1 'npx hardhat compile'` in the clone after touching a file. Compilation output must be byte-identical: compare artifacts and build-info files before and after.Source: NomicFoundation/hardhat