Runner leaks ~40KB per task via per-task vm.Script/vm.createContext, OOMs at ~10k tasks (reported as exit 139)

Summary

The runner's memory grows linearly with the number of tasks executed, independent of payload size, and dies of a V8 JavaScript heap OOM at roughly 10,000 tasks. The container then exits 139, which looks like a SIGSEGV but is not — the process is fine right up to the heap limit.

We see this every ~3 hours under load on a self-hosted deployment. Five crashes, all identical.

The exit code is misleading

The kernel logs a general protection fault in libc:

traps: node[...] general protection fault ip:... in libc.so.6[...+156000]

The faulting file offset was byte-identical across all five crashes. Symbolized against glibc 2.36 (build-id 6196744a316dbd57c0fd8968df1680aac482cec4):

0002639f T abort@@GLIBC_2.2.5
   2650f:	f4                   	hlt        <-- abort+0x170, ABORT_INSTRUCTION

So this is glibc's abort() reaching its final hlt, not memory corruption. hlt is privileged, so ring-3 execution raises #GP.

abort() falls through to hlt because node runs as PID 1 of the container's PID namespace with no init process, and the kernel discards default-action signals sent to a namespace's init — so raise(SIGABRT) returns instead of terminating. Confirmed from /proc/1/status: SigBlk: 0 and SIGABRT absent from SigCgt (nothing is catching it).

The actual cause is visible only in the container log:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
[1:0x...] 11442996 ms: Scavenge 503.9 (520.6) -> 503.0 (524.9) MB, pooled: 0 MB
 6: 0x14aefdd v8::internal::MinorGCJob::Task::RunInternal() [node]

Adding an init process (or --abort-on-uncaught-exception handling) would make this report as 134 and stop it being mistaken for a native crash. We lost several hours to that.

It is a per-task retention leak, not payload size

Two independent containers, same rate:

Evidence that it is retention rather than concurrency or payload:

• Max concurrent in-flight tasks was only 8–9. Eight small tasks cannot hold 500 MB live.
• When the task rate dropped ~30x (2,834/hr → 23/hr), the working set stayed at 535–595 MiB and never recovered.
• GC log shows scavenges reclaiming ~1 MiB from a 503 MiB old space, average mu = 0.890 — live retained objects, not allocation churn.
• The tasks in our case are tiny: the action returns a few-hundred-byte descriptor object and never touches file bytes.

Memory pressure from the cgroup is ruled out: container limit 1024 MiB, peak working set 643 MiB, container_memory_failcnt 0, oom_events_total 0, cgroup memory.events all zero. It dies at 61% of the cgroup limit because V8's own heap ceiling is 524 MiB.

Suspected mechanism

packages/runner/lib/exec.ts creates a new vm.Script and a new context per task:

const script = new vm.Script(wrappedCode, { ... });   // ~line 106
const context = vm.createContext(sandbox, { ... });   // ~line 144
const scriptExports = script.runInContext(context);   // ~line 150

This is a known Node/V8 retention pattern — compilation caches and contextified objects are not reclaimed until a last-resort GC:

• https://github.com/nodejs/node/issues/40014
• https://github.com/nodejs/node/issues/3113
• https://joyeecheung.github.io/blog/2023/12/30/fixing-nodejs-vm-apis-1/

The returned output is constructed inside the sandbox context and handed across the boundary, which would pin that context for as long as anything references it. The fatal error fires inside MinorGCJob::Task::RunInternal → Heap::CollectGarbage, i.e. during a scavenge, before a full last-resort GC can run.

This is still present on main as of 0.71.6.

We have not captured a heap snapshot (it needs a signal or restart on a live pod), so the specific retaining edge is unconfirmed — happy to gather one if that would help.

The runner's own memory monitor cannot see this

packages/runner/lib/monitor.ts compares process.memoryUsage().rss against process.constrainedMemory() — the cgroup limit (1024 MiB here). At death RSS was ~625 MiB = 61%, below the warning threshold, so Memory usage is high never logged once across the whole crashed lifetime.

The process dies of the V8 heap ceiling (heap_size_limit, 524 MiB), not the cgroup ceiling. Comparing against v8.getHeapStatistics().heap_size_limit would make this observable. Note also that raising the container limit raises the derived heap limit with it, so the monitor stays blind at any size.

Environment

• Nango 0.70.8 (image nangohq/nango:managed-1.5.11-0.70.8-...), self-hosted on Kubernetes
• Node v22.22.2, Debian 12 bookworm, glibc 2.36-9+deb12u13
• Runner replicas: 1, memory limit 1Gi, no NODE_OPTIONS set
• Workload: ~96% one action type, small JSON in and out

Suggested directions

1. Reuse a single vm.Context / compiled vm.Script per integration-script rather than per task, or explicitly dispose of the context after each run.
2. Have the memory monitor compare against v8.getHeapStatistics().heap_size_limit, not constrainedMemory().
3. Run the runner under an init process so an abort reports 134 instead of 139.

Happy to test a patch against our workload — we reproduce this reliably every ~10k tasks.