#15616·activepieces

[BUG] Worker crashes with spawn E2BIG after ~6k jobs: SANDBOX_PROCESS leaks one isolate zombie per job under node-as-PID-1, si.processes() throws uncatchably

Author: majewskibartoszCreated Sep 17, 2026Updated Sep 17, 2026
Labels🐛 bug🛠️ area/infrastructure🛟 support💫 priority

Problem

Split AP_CONTAINER_TYPE=WORKER container with AP_EXECUTION_MODE=SANDBOX_PROCESS dies every ~6,000 sandbox jobs:

Error: spawn E2BIG
    at ChildProcess.spawn (node:internal/child_process:421:11)
    at exec (node:child_process:236:25)
    at .../[email protected]/lib/processes.js:840:19

Uncaught, exit code 1. Present since 0.85.4 (#13497), reproduced live on 0.88.4-hotfix.3 and main (0.91.0 image). Not a memory issue.

Ref: Pylon 6015

Root cause

Two defects stack:

  1. Zombie leak, one per sandbox job. docker-entrypoint.sh does exec node … so the worker is PID 1 (no init). sandbox-manager.ts release()invalidate()sandbox.ts killProcess()treeKill(pid, 'SIGKILL') kills the isolate box child first, then the isolate keeper. Killed keeper never wait()s; the dead child is reparented to PID 1; libuv only reaps its own handles. Zombie forever. Live: 21 webhook runs → 42 zombies (comm=isolate, ppid 1).
  2. Uncatchable throw in telemetry. worker.ts refreshSandboxInfo (every 15 s) → system-usage.ts getProcessTreeMemoryBytesByPidssi.processes(). That builds sh -c "cat /proc/stat…;cat /proc/<pid>/stat;…" for every process incl. zombies. Past 131,072 bytes (MAX_ARG_STRLEN) Node throws spawn E2BIG synchronously inside the first exec callback (E2BIG is not in Node's async-emit errno list). tryCatch around the promise cannot see it; worker has no uncaughtException handler.

Upstream systeminformation never fixed it (issues 968, 996 closed by raising maxBuffer; 5.33.11 still concatenates).

Not affected: WORKER_AND_APP (sh is PID 1 and reaps), anything with an init. SANDBOX_CODE_ONLY + reuse leaks only on kills (slow trickle).

Fix

Both, tested live on the same stack (60 sandbox jobs each → 0 zombies; details + patch in the linked report):

  1. Dockerfile: apt-get install -y --no-install-recommends tini in runtime stage, ENTRYPOINT ["tini", "--", "./docker-entrypoint.sh"]. Also reaps deno / node --eval orphans killed on timeout.
  2. packages/server/utils/src/system-usage.ts: replace si.processes() with a direct /proc/<pid>/status read (PPid, VmRSS), keep systeminformation only off Linux. No child process, no argv limit. Survived 14,737 procs / 7,000 zombies where si.processes() threw E2BIG. 23 unit tests, build, lint pass.

Optional: in isolate mode kill box children and let the keeper exit on its own before SIGKILL. Unnecessary once (1) ships.

Customer mitigation (verified): Kubernetes shareProcessNamespace: true on the worker pod (pause reaps), or sh -c './docker-entrypoint.sh & p=$!; trap "kill -TERM $p" TERM INT; wait $p; wait $p' command override, or compose init: true.

Source: activepieces/activepieces