#3717 Possible 1.22.2 CE working-set regression causing sustained swap/paging on 1 GB VPS

Describe the Bug

After upgrading Pangolin Community Edition from 1.21.1 to 1.22.2 on the same VPS and configuration, host resource usage changed substantially.

Before the upgrade, the VPS had been stable after unrelated monitoring optimizations. After the 1.22.x generation was deployed, the host entered sustained memory pressure with much higher swap activity, major page faults, CPU system time, and I/O wait.

Direct cgroup and /proc measurements point to the Pangolin application process as the primary source of the new memory pressure rather than filesystem cache.

At one sampled point, the Pangolin application process had:

RssAnon: 201,088 KiB
VmSwap: 106,112 KiB

That is approximately 300 MiB of combined anonymous resident + swapped memory.

In one 10-minute pressure window:

host major faults: 120,882
Pangolin major faults: 74,938 (~62% of host total)
Pangolin anonymous memory increased by ~95.34 MiB
CrowdSec simultaneously lost ~56.53 MiB of file cache and performed heavy file refault/read activity

This looks like Pangolin expanding its private working set, which forces this small VPS into paging. Other services then lose useful cached pages and reread them, amplifying CPU and disk I/O.

I am not claiming that this is proven to be a simple monotonic memory leak. After a Pangolin-only restart, an initial 10-minute warm-up showed substantial growth, but a subsequent warmed 30-minute sample largely plateaued. The behavior appears bursty/workload-triggered rather than a simple linear leak.

The configuration that had performed acceptably before the upgrade did not materially change.

Six Pangolin resource-target health checks were configured at 5-second intervals both before and after the 1.22 upgrade. A pre-1.22 database backup confirms the same checks and cadence existed before the regression.

A matched 600-second A/B test showed that enabling those six checks materially increases paging:

host swap-in: ~31% higher
host swap-out: ~34% higher
host major-fault rate: ~25% higher
Pangolin anonymous-refault rate: ~42% higher
Pangolin major-fault rate: ~32% higher

I have therefore disabled those six target health checks as a mitigation. This helps, but they cannot independently explain the regression onset because the same checks were already enabled before Pangolin 1.22.

Longer-term host statistics also show a clear change:

Metric	Before regression (Sep 1-3)	Regression (Sep 6-7)
CPU busy	~12-13%	~19-21%
swap-in	~13-15 pages/s	~163-184 pages/s
swap-out	~7 pages/s	~86-109 pages/s
major faults	~10-13/s	~133-145/s

Additional investigation did not identify an obvious external trigger:

the measured pressure window had less Traefik request traffic than its preceding control window
no repeating high-volume client pattern was found
Pangolin request-audit rows did not correlate with the measured pressure interval
available Pangolin logs showed no Newt endpoint-roaming/reconnect churn
no OOM condition is required for the regression to occur
vm.swappiness remained unchanged at 60

This may be related conceptually to #2120 and #2134, but this report is from Pangolin 1.22.2 Community Edition and includes direct application/cgroup paging attribution.

The retained previous Pangolin image is 1.21.1 and uses Node 24.18.0. The current 1.22.2 image uses Node 24.18.1. I consider that patch-level Node change a secondary variable, not a demonstrated cause.

The leading hypothesis is a Pangolin 1.22 application/dependency working-set regression or behavior change. I do not currently have evidence identifying the exact code path.

Environment

OS Type & Version: Ubuntu 24.04.4 LTS, kernel 6.8.0-138-generic, x86_64
Pangolin Version: 1.22.2
Edition (Community or Enterprise): Community
Gerbil Version: exact semantic version not captured
Traefik Version: 3.7.13
Newt Version: not captured for this report
Client Version: N/A

Additional environment details:

Previous known-good Pangolin Version: 1.21.1
VPS: 1 vCPU, approximately 961 MiB RAM, 1 GiB swap
Docker: 29.7.2
Docker Compose: 5.4.0
Pangolin 1.22.2 Node Version: 24.18.1
Pangolin 1.21.1 Node Version: 24.18.0
CrowdSec: 1.8.1-909b5157
Host vm.swappiness: 60

To Reproduce

The exact workload that triggers each Pangolin memory-growth burst is not yet identified, but the regression is observable in normal operation.

Run Pangolin CE 1.21.1 on a roughly 1 GB VPS with normal public resources and resource-target health checks.
Establish a stable host CPU/paging baseline.
Upgrade Pangolin to 1.22.2 without materially changing the resource configuration.
Monitor host pswpin, pswpout, major faults, CPU/I/O wait, and Pangolin cgroup memory/swap.
Inspect the Pangolin application process with /proc/<pid>/status or smaps_rollup; significant private/anonymous resident + swapped memory is observable.
If resource-target health checks are enabled, compare equivalent intervals with them enabled and disabled. In this environment they significantly amplify the paging, although they were already present before the 1.22 upgrade.

A Pangolin-only restart temporarily reduces host swap usage, but the higher memory-pressure behavior can return afterward.

I can provide additional sanitized cgroup, /proc, sysstat, or A/B measurements if there is a specific measurement that would help identify the responsible 1.22 code path.

AI Disclosure

AI was used to help organize and draft this report from diagnostic measurements I collected. The numerical observations, version information, test results, and conclusions were reviewed against the measurements before submission. AI was not used as evidence for the reported behavior.

Expected Behavior

On the same configuration and workload, Pangolin 1.22.x should maintain a reasonably similar steady-state working set to 1.21.1 and should not push a previously stable 1 GB VPS into sustained swap/refault cycles.

Resource-target health checks should also not increase host paging and major-fault activity by the observed magnitude.