Windows: working set grows to tens of GB from memory-mapped column files and is only reclaimed lazily — please consider an opt-in working-set trim

QuestDB version

10.0.1

OS, in case of Docker specify Docker and the Host OS

Windows 11 x64 (build 26200), QuestDB installed as a Windows service (running as LocalSystem), bare metal, 64 GB RAM.

Description

QuestDB's storage engine memory-maps column files. On Windows, pages touched by queries stay in the process working set, and the OS only trims them lazily (minutes-scale idle trimming, or under system memory pressure). During sustained analytical workloads this makes the java process RSS grow essentially unboundedly, even though almost none of it is private memory:

• Workload: batches of long full-table scans / joins over a ~12M-row daily table, running continuously for ~1 hour (factor research pipeline; hundreds of sequential heavy queries).
• Observed: java RSS grew 3.3 GB -> 32.8 GB in ~35 minutes while PrivateMemorySize64 stayed at 4.4–9.8 GB — i.e. the working set is dominated by file-backed pages (memory-mapped column data), not allocations QuestDB owns.
• Consequence: system-wide available memory dropped to 5.6 GB on a 64 GB host. Co-located processes get squeezed; the machine behaves as if QuestDB were leaking, even though the memory is effectively a read cache Windows refuses to reclaim promptly.

Relief experiment

Calling EmptyWorkingSet (psapi) on the QuestDB java process gives instant relief, with no observable impact on the running workload:

Metric	Before	After
java working set	28.1 GB	1.9 GB
System available memory	5.6 GB	35.4 GB

The query batch kept running at the same throughput during and after the trim (subsequent per-query timings unchanged) — consistent with the trimmed pages being served back from the standby list as cheap soft faults. Reproduced 3 times (e.g. 28.1 -> 1.9 GB, 14.8 -> 1.5 GB).

Why I think this deserves upstream attention

1. On Linux, mapped file pages are reclaimed much more aggressively under pressure, so this failure mode is mostly invisible there — Windows deployments silently degrade instead.
2. The pages involved are reclaimable cache, so an OS-level trim is safe by construction; but a naive periodic trim would discard QuestDB's own read cache for no reason. Hence an opt-in / policy-based mechanism rather than an unconditional one.

Suggestions (any of these would help)

• A server config knob for Windows, e.g. cairo.working.set.trim=off|idle|interval:<n>, calling SetProcessWorkingSetSize(handle, -1, -1) / EmptyWorkingSet on an idle timer (no queries in flight) rather than on a blind clock;
• Or a management endpoint / console command to trigger a trim on demand, so operators can wire it into their own monitoring;
• Or at minimum, a docs section describing the Windows working-set behavior and the manual mitigation.

Relation to existing issues

Different axis from #7592 and #4941: those concern the RSS_MEM_USED limiter accounting (tagged native mallocs). Here the issue is purely OS-level working-set growth from file-backed mapped pages, which the limiter does not even count.

To reproduce

1. Windows host, QuestDB as a service, a table of at least ~10M rows.
2. Run a continuous batch of full-scan queries (e.g. joins + SAMPLE BY over the whole table) for ~30–60 min.
3. Watch the java process RSS vs. private memory (Task Manager / Get-Process): RSS climbs to tens of GB, private stays flat.
4. Optional: run [psapi]::EmptyWorkingSet($proc.Handle) and observe RSS drop to ~2 GB with no query slowdown.