self-hosted backend memory after `convex deploy`: two owners, neither released
Author: etylsarinCreated Sep 17, 2026Updated Sep 17, 2026
Setup. Self-hosted convex-local-backend, release precompiled-2026-09-15-a35911b (also reproduced on the 2026-03-24 build), SQLite, one deployment per container, cgroup v2 memory cap 1536 MiB, REUSE_ISOLATES default (true), all knobs default. Deployment: ~140 modules, 24 of them "use node".
Observation. A 15-second per-process trace inside the container through one convex deploy with no code change:
| time | phase | cgroup anon | convex-local-backend RSS |
node local.cjs RSS |
|---|---|---|---|---|
| +0 | fresh restart | 23 MiB | 78 | — |
| +1 min | idle | 121 | 186 | — |
| +4 min | 42 concurrent /api/query (Next.js prerender) |
671 | 738 | — |
| +6 min | evaluate_push … finish_push |
1132 | 882 | 376 |
| +23 h | idle, no traffic, no deploys | flat | flat | flat |
- The backend process grows ~550 MiB when 42 distinct queries execute concurrently (8 funrun isolate workers were created), and ~145 MiB more on the push. Nothing comes back after
ISOLATE_IDLE_TIMEOUT_SECONDS(600) orISOLATE_MAX_LIFETIME_SECONDS(3600): a 23-hour idle window stayed flat. CappingFUNRUN_CODE_CACHE_SIZE/FUNRUN_MODULE_CACHE_SIZE/SOURCE_MAP_CACHE_MAX_SIZE_BYTESto 100/50/20 MB andMAX_ISOLATE_WORKERSto 16 changed nothing (1004 MiB vs 883 with defaults on the next identical push), so the level is not the caches. - The local node executor (
crates/node_executor/src/local.rs) is spawned by the first"use node"invocation, loads every node bundle during push analysis, and never exits (kill_on_droponly; respawned only on connection failure orexitingProcess). It sits at 300–375 MiB for the life of the backend process. On a production deployment with no deploy that day it lived 9 h at 288 MiB after a single daily"use node"cron.
Questions / asks.
- Is isolate memory expected to return to the OS on worker recreation? If the intent is yes, the 23-hour flat curve suggests it does not for
REUSE_ISOLATES=true; if no, a knob to bound per-worker retention would help self-hosters on small boxes. - A lifetime or RSS threshold for the local node executor (recycle after N invocations / N MB) would bound the second owner; today the only release is a container restart.
Related: #312 (TooMuchMemoryCarryOver). Happy to attach the raw traces and memory.stat snapshots.
Source: get-convex/convex-backend