Self-hosted: in-process searchlight disk cache and segment LRUs are uncoordinated — RAM growth from deleted-but-mapped segments, meta.json watcher log flood, and ENOENT query failures

Environment

• Hosting: Railway (self-hosted), single instance, persistent volume at /convex/data
• Image: ghcr.io/get-convex/convex-backend:latest (currently sha256:240a2e9115dab3c13145b96532d1cc8bc63c5e2d112d95f80e592a75242bfa75)
• Resources: 32 GB cgroup memory limit
• Dataset: ~21 GB of text + vector index segments (one large vector index with ~650k chunks/embeddings, several smaller text + vector indexes)

Summary

In self-hosted deployments, search runs through InProcessSearcher, which layers two caches that are not coordinated with each other:

1. An archive disk cache that extracts segment archives into tmpdir/<uuid>/ and evicts by deleting extracted directories — hardcoded to 500 MiB with no env knob (crates/search/src/searcher/in_process.rs#L147).
2. Two in-memory segment LRUs (TextSegmentCache, VectorSegmentCache) that hold live tantivy::Searchers and qdrant Segments (each with its own RocksDB instance), keyed by extracted path and counted per entry (SizedValue::size() = 1, defaults 120 + 120 — segment_cache.rs, searchlight_knobs.rs#L82-L92).

Because the disk cache deletes directories that the memory LRUs still reference, any deployment whose working set exceeds 500 MiB of extracted segments (i.e. any non-trivial dataset) ends up in one of two failure modes depending on which side loses the race:

• Memory-LRU entries outlive their directories → RAM grows to the cgroup limit via deleted-but-still-mapped files, plus a tantivy file-watcher log flood.
• Re-opening an evicted segment needs a directory the disk cache already deleted → No such file or directory (os error 2) surfaces to end users as Your request couldn't be completed. Try again later. on ctx.vectorSearch / text search, and crashes VectorCompactor (this looks like the same root cause as #317).

Symptom 1: RAM grows until the cgroup limit ("memory leak")

RSS after a redeploy starts ~8 GB, plateaus 21–24 GB, and we've measured a peak of 31.7 GB against our 32 GB limit — recovered only by redeploying.

Measurements on the live instance (2 days uptime, RSS 9.1 GB of which 8.9 GB anon private dirty):

• 1,108 deleted-but-still-mapped files from the searchlight tmpdir, 14.71 GiB of mappings across 223 distinct extracted directories — which matches two full 120-entry LRUs almost exactly.
• On disk, only 9 of those directories still existed. Everything else was memory pinned by the LRUs after the disk cache had deleted the directories.
• The other backend caches (function runner, index cache) are bounded around ~1.5 GB and do not explain the growth.

Mechanism: after the disk cache evicts a directory, a refetch of the same logical segment lands under a new uuid path. Since the LRUs key by path, that becomes a new entry, and the old entry stays behind as dead weight until the 120-entry cap eventually rotates it out. In steady state both LRUs are full of mostly-dead segments whose mmaps pin deleted files.

Symptom 2: tantivy file-watcher log flood

Every cached text segment holds an IndexReader created via Index::open_in_dir(...) + index.reader() (crates/search/src/disk_index.rs#L77-L88), i.e. tantivy's default reload policy, which spawns a file watcher polling meta.json every ~500 ms. Once the directory is deleted, each of those watchers logs

WARN tantivy::directory::file_watcher: Failed to open meta file ".../tmp/.tmpXXXX/<uuid>/meta.json": Os { code: 2, kind: NotFound, ... }

twice a second, forever. We've measured a sustained 117 log lines/sec (116 unique pinned directories — the text LRU at its 120 cap), which was enough to hit Railway's log rate limit and drown out everything else. Since these segments are immutable, ReloadPolicy::Manual would eliminate both the watcher threads and the flood.

Symptom 3: ENOENT query failures / VectorCompactor crashes (likely #317)

We mitigated symptom 1 by lowering the LRU caps via the env knobs (MAX_TEXT_LRU_ENTRIES / MAX_VECTOR_LRU_ENTRIES, which also caps MAX_VECTOR_LRU_SIZE since it reads the same env var). That bounded RAM as expected — but setting them below the hot working set (we used 16/16) flipped the race to the other side: under normal evening load we started seeing intermittent

Uncaught Error: Your request couldn't be completed. Try again later.

on every search-using function, with the backend logging

ERROR ...: Caught error error ...: Orig Error: No such file or directory (os error 2). instance_name="convex_self_hosted"

An evicted segment gets re-opened while its extracted directory has already been deleted by the disk cache → ENOENT mid-query. Retries succeed because the refetch re-extracts under a new path — so it presents as flapping, load-dependent failures. Text search mostly survives directory deletion (mmap keeps the data alive); qdrant/RocksDB opens files by path, which is why the hard failures concentrate on the vector path — consistent with the VectorCompactor died: No such file or directory crash loop in #317 (whose "temp files lost on restart" hypothesis doesn't match what we observe: the deletions happen while the process is running, by the archive cache's own eviction).

Raising the knobs to 64/64 moved us to an operating point where both symptoms are quiet (bounded RAM, no eviction races at our QPS) — but that's tuning around the race, not removing it.

Suggested fixes

1. Coordinate the two caches: the disk cache should not delete an extracted directory while a memory-LRU entry still references it (refcount / delete-on-last-drop), and conversely a memory-LRU eviction should be what releases the directory. Keying the memory LRUs by segment identity instead of extracted path would also stop the dead-weight duplication.
2. ReloadPolicy::Manual in index_reader_for_directory — the segments are immutable; the default watcher only burns CPU and floods logs.
3. Make the archive disk cache size configurable (it's the only piece of this without an env knob) and/or scale it with the volume size — 500 MiB is far below any realistic self-hosted dataset.

Happy to provide more measurements (smaps/lsof snapshots, log captures) if useful.