Node heap OOM on large text asm output: two prod nodes killed by one ~13 KB source (2026-08-14)

(Filed by Claude on behalf of @mattgodbolt; follows from investigation in compiler-explorer/infra#2310.)

Summary

On 2026-08-14 ~06:46 UTC two prod nodes died within 40 s of each other with FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory. Both were the same user compiling the same ~13 KB C++ source: once with gcc 16.2 (the node OOMed, the user got a 504 after 60 s), then again with clang 22.1 after switching compiler (second node OOMed, 502). Each node went from normal to a ~2 GB heap inside that single request, so this is not a leak. These are the only two heap OOMs in the searchable log window (since ~2026-08-11), none on staging or beta.

What the logs show

ip-172-30-4-158 (i3.large, 15 GiB RAM) and ip-172-30-0-86 (m6i.large, 8 GiB RAM, 21 minutes old):

06:46:32 [1283] 3393329 ms: Scavenge (interleaved) 2041.0 (2080.9) -> 2041.0 (2085.9) MB
06:46:32 [1283] 3396414 ms: Mark-Compact (reduce) 2044.0 (2085.9) -> 2044.0 (2082.9) MB, 2170.70 ms
06:46:32 FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
06:46:32  9: v8::internal::Runtime_AllocateInYoungGeneration
06:47:08 systemd: compiler-explorer.service: Main process exited, code=killed, status=6/ABRT

Both nodes hit the V8 default old-generation limit (~2 GB) regardless of RAM: nothing sets --max-old-space-size or NODE_OPTIONS (infra/init/start.sh runs node bare; init/compiler-explorer.service has no heap or Restart= settings). The app log goes silent 25-45 s before the crash, i.e. the main thread was busy parsing, which is also why /healthcheck stalled and the ALB pulled the node.

Likely code path

Text asm is only bounded by max-asm-size=67108864 (64 MiB, etc/config/compiler-explorer.amazon.properties), checked in lib/base-compiler.ts. Below that:

• processAsm -> AsmParser.processAllLines (lib/parsers/asm-parser.ts) builds one {text, source, labels} object per line with no line cap. The binary/objdump path has maxAsmLines=5000; the text path has nothing equivalent.
• then the demangler walks the result, cachePut JSON.stringifys it (lib/compilation-env.ts), and res.send stringifies it again (lib/handlers/compile.ts).

Measured locally with the real AsmParser on synthetic gcc -S -g output: 16 MB -> ~300 MB heap, 32 MB -> ~600 MB (default filters) / ~985 MB (filters off), 60 MB -> ~1.1 GB, before the two JSON copies. So ~40-64 MB of asm, well under the cap, exhausts a 2 GB heap, and a 13 KB source producing that with both gcc and clang is plausible (large constexpr tables / template fan-out; -g doubles the line count with .locs). Parse time for 60 MB was ~20 s locally, consistent with the silent gap before the crash.

The kernel cgroup OOM-kills of cc1plus/clang++ at 1.25 GiB in the same logs are the nsjail compile limit working as intended and are unrelated.

Suggested fixes

1. Cap text asm by line count in processAllLines, the same way processBinaryAsm does (with the existing [truncated; too many lines] handling downstream), and/or drop max-asm-size to ~8-16 MiB. This is the actual fix; the UI cannot render that much anyway.
2. Set --max-old-space-size explicitly in infra/init/start.sh (e.g. 4096 on 8 GiB boxes; the nsjail compile cgroup is 1.25 GiB x maxConcurrentCompiles=2, so there is room). On its own this only moves the threshold; do it with (1).
3. Restart=on-failure / RestartSec=2 in infra/init/compiler-explorer.service: the process was dead for 30-70 s before the ASG acted, and replacement costs a fresh instance plus discovery.
4. Log compiler id and asm byte size / parse time at info level so the next occurrence names the request; the OOM itself never reaches Sentry.