self-hosted backend memory after `convex deploy`: two owners, neither released

Author: etylsarinCreated Sep 17, 2026Updated Sep 17, 2026

Setup. Self-hosted convex-local-backend, release precompiled-2026-09-15-a35911b (also reproduced on the 2026-03-24 build), SQLite, one deployment per container, cgroup v2 memory cap 1536 MiB, REUSE_ISOLATES default (true), all knobs default. Deployment: ~140 modules, 24 of them "use node".

Observation. A 15-second per-process trace inside the container through one convex deploy with no code change:

time phase cgroup anon convex-local-backend RSS node local.cjs RSS
+0 fresh restart 23 MiB 78
+1 min idle 121 186
+4 min 42 concurrent /api/query (Next.js prerender) 671 738
+6 min evaluate_pushfinish_push 1132 882 376
+23 h idle, no traffic, no deploys flat flat flat
  1. The backend process grows ~550 MiB when 42 distinct queries execute concurrently (8 funrun isolate workers were created), and ~145 MiB more on the push. Nothing comes back after ISOLATE_IDLE_TIMEOUT_SECONDS (600) or ISOLATE_MAX_LIFETIME_SECONDS (3600): a 23-hour idle window stayed flat. Capping FUNRUN_CODE_CACHE_SIZE/FUNRUN_MODULE_CACHE_SIZE/SOURCE_MAP_CACHE_MAX_SIZE_BYTES to 100/50/20 MB and MAX_ISOLATE_WORKERS to 16 changed nothing (1004 MiB vs 883 with defaults on the next identical push), so the level is not the caches.
  2. The local node executor (crates/node_executor/src/local.rs) is spawned by the first "use node" invocation, loads every node bundle during push analysis, and never exits (kill_on_drop only; respawned only on connection failure or exitingProcess). It sits at 300–375 MiB for the life of the backend process. On a production deployment with no deploy that day it lived 9 h at 288 MiB after a single daily "use node" cron.

Questions / asks.

  • Is isolate memory expected to return to the OS on worker recreation? If the intent is yes, the 23-hour flat curve suggests it does not for REUSE_ISOLATES=true; if no, a knob to bound per-worker retention would help self-hosters on small boxes.
  • A lifetime or RSS threshold for the local node executor (recycle after N invocations / N MB) would bound the second owner; today the only release is a container restart.

Related: #312 (TooMuchMemoryCarryOver). Happy to attach the raw traces and memory.stat snapshots.

Source: get-convex/convex-backend