Self-host: every HTTP request retains ~80 MiB per 1000 requests; Kubernetes health probes alone OOM-kill the container

Summary

Running the self-host image (Dockerfile.selfhost, AUTH_MODE=local_noauth) under a 2 GiB memory limit, the container's working set grows with every HTTP request and idle never returns it. In our Kubernetes deployment the readiness and liveness probes on GET /api/health (480 requests/hour, no other traffic) took the pod from ~600 MiB to the 2 GiB limit in about 33 hours, and it was OOM-killed on that cycle for days. The same happens on v0.1.6 and v0.1.8, and in D1 mode as well as Postgres mode.

Measurement (2026-09-14)

Two containers side by side on the same host, each docker run --memory=2g, AUTH_MODE=local_noauth, CLOUDFLARE_INCLUDE_PROCESS_ENV=true, DATABASE_PROVIDER=postgres with a local postgres:16, no DataForSEO key. GET /api/health at 8 requests/second for 12 minutes from a small Python loop; docker stats memory sampled every 30 seconds.

Control: the v0.1.8 image in default D1 mode (no DATABASE_PROVIDER, so withPgClient is a no-op and no Postgres client is created) still retained 73 MiB per 1000 requests under the same load, with no recovery on idle.

cgroup memory.stat anon tracked the docker stats figure within ~15 MiB throughout, so this is anonymous memory held by the process, not page cache.

What it is not

• Not Postgres connections. A client is created per request (src/server.ts wraps every fetch in withPgClient, and src/db/pg/client.ts deliberately never calls sql.end() because Hyperdrive reclaims the socket at invocation end). Under the test pg_stat_database.sessions rose by one per request, but numbackends stayed 0 at every sample, so the connections close promptly. Comparing Postgres mode (89-92 MiB/1000) with D1 mode (73 MiB/1000), the unclosed client accounts for roughly 6-11 MiB per 1000 requests; the rest is elsewhere.
• Not the site-audit or SAM paths: /api/health only calls getSelfHostSetupStatus() (src/server/lib/setup-status.ts), whose database check is a single select count(*) from projects.
• Not a v0.1.7/v0.1.8 regression: the diff between v0.1.6 and v0.1.8 over src/routes/api/health.ts, src/server/lib/setup-status.ts, src/db/pg/client.ts and src/db/provider.ts is empty, and the two builds measure the same.

So the retention appears to be in the workerd request path of the self-host server (the vite preview + workerd serve process that docker-entrypoint.sh starts), roughly 70-80 MiB per 1000 requests regardless of what the route does.

Reproduce

docker build -f Dockerfile.selfhost -t open-seo-selfhost .
docker run -d --name oseo --memory=2g -p 127.0.0.1:3401:3001 \
  -e AUTH_MODE=local_noauth -e CLOUDFLARE_INCLUDE_PROCESS_ENV=true \
  -e ALLOWED_HOST=localhost -e PORT=3001 open-seo-selfhost
# wait for /api/health to report ok, note docker stats, then:
for i in $(seq 1 3000); do curl -s -o /dev/null http://127.0.0.1:3401/api/health; done
docker stats --no-stream oseo   # roughly +200-250 MiB, and it does not come back

Environment

• Image: Dockerfile.selfhost at v0.1.6 and at v0.1.8 (plus our self-host patches for Postgres, which do not touch the request path; the D1-mode control run has no Postgres code active)
• Host: Docker 29.1, Linux 6.8, x86_64; in production Kubernetes with a 2 GiB limit
• workerd 1.20260625.1, Node 22 in the image

Impact and workaround

Any periodic HTTP prober (Kubernetes probes, uptime checks) alone will OOM the container on a cycle. We halved the probe rate and switched liveness to a TCP probe, which only stretches the cycle. A fix would need to find what the per-request path retains; if it is workerd-specific, the self-host entrypoint may need a different serving mode than vite preview.

Happy to run further measurements or test a patch.