[BUG] Worker crashes with spawn E2BIG after ~6k jobs: SANDBOX_PROCESS leaks one isolate zombie per job under node-as-PID-1, si.processes() throws uncatchably
Problem
Split AP_CONTAINER_TYPE=WORKER container with AP_EXECUTION_MODE=SANDBOX_PROCESS dies every ~6,000 sandbox jobs:
Error: spawn E2BIG
at ChildProcess.spawn (node:internal/child_process:421:11)
at exec (node:child_process:236:25)
at .../[email protected]/lib/processes.js:840:19Uncaught, exit code 1. Present since 0.85.4 (#13497), reproduced live on 0.88.4-hotfix.3 and main (0.91.0 image). Not a memory issue.
Ref: Pylon 6015
Root cause
Two defects stack:
- Zombie leak, one per sandbox job.
docker-entrypoint.shdoesexec node …so the worker is PID 1 (no init).sandbox-manager.tsrelease()→invalidate()→sandbox.tskillProcess()→treeKill(pid, 'SIGKILL')kills the isolate box child first, then the isolate keeper. Killed keeper neverwait()s; the dead child is reparented to PID 1; libuv only reaps its own handles. Zombie forever. Live: 21 webhook runs → 42 zombies (comm=isolate, ppid 1). - Uncatchable throw in telemetry.
worker.tsrefreshSandboxInfo(every 15 s) →system-usage.tsgetProcessTreeMemoryBytesByPids→si.processes(). That buildssh -c "cat /proc/stat…;cat /proc/<pid>/stat;…"for every process incl. zombies. Past 131,072 bytes (MAX_ARG_STRLEN) Node throwsspawn E2BIGsynchronously inside the first exec callback (E2BIG is not in Node's async-emit errno list).tryCatcharound the promise cannot see it; worker has nouncaughtExceptionhandler.
Upstream systeminformation never fixed it (issues 968, 996 closed by raising maxBuffer; 5.33.11 still concatenates).
Not affected: WORKER_AND_APP (sh is PID 1 and reaps), anything with an init. SANDBOX_CODE_ONLY + reuse leaks only on kills (slow trickle).
Fix
Both, tested live on the same stack (60 sandbox jobs each → 0 zombies; details + patch in the linked report):
Dockerfile:apt-get install -y --no-install-recommends tiniin runtime stage,ENTRYPOINT ["tini", "--", "./docker-entrypoint.sh"]. Also reaps deno /node --evalorphans killed on timeout.packages/server/utils/src/system-usage.ts: replacesi.processes()with a direct/proc/<pid>/statusread (PPid,VmRSS), keep systeminformation only off Linux. No child process, no argv limit. Survived 14,737 procs / 7,000 zombies wheresi.processes()threw E2BIG. 23 unit tests, build, lint pass.
Optional: in isolate mode kill box children and let the keeper exit on its own before SIGKILL. Unnecessary once (1) ships.
Customer mitigation (verified): Kubernetes shareProcessNamespace: true on the worker pod (pause reaps), or sh -c './docker-entrypoint.sh & p=$!; trap "kill -TERM $p" TERM INT; wait $p; wait $p' command override, or compose init: true.
Source: activepieces/activepieces