Make zram swap size proportional to node RAM (currently hardcoded to 1 GiB)

What I'd like:

That the zram swap device introduced in core-kit#590 should be sized such that it cannot mask kubelet's memory.available eviction signal — i.e. sized relative to the node's RAM and to the configured eviction thresholds, rather than as a fixed constant.

What actually happened:

prepare-swap.service creates a fixed 1 GiB device on every node, regardless of effective RAM and regardless of the eviction thresholds in effect:

ExecStart=/usr/sbin/zramctl /dev/zram0 --size 1G

Verified unchanged on 1.63.0 (16 GiB r7g.large):

# cat /sys/block/zram0/disksize
1073741824
# cat /proc/sys/vm/swappiness /proc/sys/vm/page-cluster
200
0

Because swapped-out anonymous pages leave workingSet, and kubelet computes memory.available = capacity − workingSet, the device is a blind spot in the eviction signal of up to its own size. When it is large relative to the eviction threshold, neither eviction nor the OOM killer fires and the node thrashes indefinitely instead of shedding load.

The default configuration cannot trigger hard eviction

This is the part I'd most like maintainers to look at, because it is not a small-instance edge case — it affects every node running stock settings. Unless of course, my analysis is wrong ;-).

The kubelet config template emits eviction thresholds only if the user set them (kubelet-config):

{{#if settings.kubernetes.eviction-hard}}
evictionHard:
  {{#each settings.kubernetes.eviction-hard}}
  {{@key}}: "{{this}}"
  {{/each}}
{{/if}}

Bottlerocket ships no default, so a stock node falls back to kubelet's own built-in default — memory.available<100Mi, absolute — and no soft thresholds at all.

Now consider a full zram device. Its pages are compressed into zsmalloc memory, which is unaccounted kernel memory: it occupies physical RAM but appears in nobody's workingSet. So with a device of size Z and compression ratio r, kubelet's reported memory.available has a hard floor of roughly Z / r, and cannot go below it no matter how much memory the workload demands:

Every one of those is above the 100 MiB default threshold. So on a stock Bottlerocket node with a saturated zram device, memory.available never reaches evictionHard, at any instance size — 2 GiB or 768 GiB alike.

Two defaults that ship together are mutually incompatible: swap-on-by-default and kubelet's 100 MiB absolute hard-eviction default. Whichever way this issue is resolved, I think those two need to be reconciled.

Why a fixed size is the wrong shape of knob

For users who do configure percentage thresholds, the failure becomes size-dependent instead of universal. A threshold is only reachable if its band exceeds the device. With our thresholds (hard 5% / soft 15%):

(Nominal RAM; kubelet's capacity is ~4–6% lower.)

The core point is scale invariance. A fixed 1 GiB device is safe only above some node size — ~20 GiB for a 5% hard threshold, ~6.7 GiB for a 15% soft one. A device sized as a fraction of the threshold is safe at every size, which is the property a node OS needs, since it cannot know what instance types or thresholds its users will run.

Note also that #4735's own validation was performed on a 7.6 GiB node — right at the boundary where the soft band stops exceeding the device.

What this looked like in production

A c5a.large (3.7 GiB) node went into permanent reclaim thrash. kubelet reported 46% memory available while only 91 MiB was actually free, at ~29,700 page ops/s. No eviction and no OOM kill ever occurred; the node simply degraded until we replaced it.

• memory PSI full avg300: 21.71%; io PSI full: 6.99%
• zram: 318 MiB of anon parked; compression ratio 3.0–4.1× (mm_stat)
• page cache: ~1.8 GiB, hot and constantly refaulting
• A healthy node of identical uptime and instance family: pswpin/pswpout = 0
• Collateral damage: our monitoring agent's probes timed out → exitCode 137, 1524 restarts. That is what led us to investigate.

To be precise about causation: on this node the larger share of the memory.available inflation was inactive page cache, not zram — that is the long-standing kubernetes#43916 blind spot, which is not Bottlerocket's doing. zram contributed the smaller share directly.

Its real contribution was to make the state stable. With vm.swappiness = 200 and a 1 GiB absorber, the kernel had somewhere to keep putting anon pages, so it ping-ponged indefinitely rather than being forced into reclaim that would have resolved the situation one way or the other. The two blind spots also compose — which is the main reason the proposal below sizes the device at a fraction of the threshold rather than up to it.

After moving the same workloads to 16 GiB nodes (no other change): memory PSI full 0.00 (total = 0), zram used 0, 0 restarts.

We worked around this by enforcing a ≥16 GiB floor on all node pools — a blunt and expensive workaround for what is really a sizing constant, and one that is unavailable to anyone whose workloads legitimately belong on small nodes.

Existing workarounds are all unpleasant

• There is no supported setting to resize or disable the device. settings.kubernetes.memory-swap-behavior (v1.42.0) governs pod swap only and has no effect on the host device.
• The workarounds circulating in #4075 are a bootstrap container running swapoff / zramctl -r / modprobe -r zram (with a retry loop for Device or resource busy), or vm.swappiness=60 via EC2NodeClass userData. Both fight the OS rather than configure it, and the latter does not address the accounting blind spot at all.
• Upstream Kubernetes will not solve this: making the eviction manager swap-aware is an explicit Non-Goal of KEP-2400, kubernetes#129578 was closed unmerged, and no PSI-based eviction signal is proposed.

Proposed change

[!NOTE]
Maybe there are better solutions to this problem than what I propose here (I am really not a subject matter exper neither on ZRAM nor on kubelet's eviction feature). So, please do not hesitate to bring up more suitable alternatives.

The invariant

Size the device so that a full device still leaves memory.available able to cross the tightest threshold that is meant to fire:

Z  ≤  α · T        where  T = min over the configured memory.available
                              eviction thresholds (hard and soft),
                              resolved to bytes against MemTotal

α = 0.5 is the value I'd suggest. The floor on memory.available is Z / r, so even at the worst case r = 1 (incompressible data, which zram stores raw) α = 0.5 leaves a 2× margin — and that margin is what absorbs the page-cache blind spot the two effects compose into.

Taking min(hard, soft) rather than one or the other is what makes both thresholds keep working, which is the property we actually want.

Option A — derive the size from the thresholds

Z = min( α · T , Z_max )        Z_max = 4 GiB

With hard 5% / soft 15%, so T = 5% · MemTotal:

The constant is wrong in both directions — it defeats eviction below ~32 GiB and under-provisions the absorber above ~64 GiB. That asymmetry is, I think, the clearest argument that this should not be a constant.

If the computed size falls below a small floor (~64 MiB), I'd suggest simply not enabling swap: at that point the device cannot help, and silently clamping it upward would reintroduce the bug.

Implementation. The thresholds are already settings, so this needs no new API surface:

1. Render the existing values into an env file via the normal template mechanism, e.g. /etc/zram.env:
   复制

   EVICTION_MEMORY_AVAILABLE_HARD={{#if settings.kubernetes.eviction-hard}}...{{/if}}
   EVICTION_MEMORY_AVAILABLE_SOFT={{#if settings.kubernetes.eviction-soft}}...{{/if}}

2. Replace the literal --size 1G with a ~25-line shell helper that reads that file plus MemTotal from /proc/meminfo, handles both N% and NMi/NGi forms, and computes Z. No new Handlebars helper needed — the arithmetic stays in shell, the template only passes the raw strings through.
3. Reorder the unit. This is the one real cost, and it is small:
   diff复制

   -DefaultDependencies=no
   [email protected]
   [email protected] dev-zram0.device
   [email protected]
   [email protected] dev-zram0.device settings-applier.service
   
    [Install]
   -WantedBy=sysinit.target
   +RequiredBy=preconfigured.target

   settings-applier.service (thar-be-settings --all) is where every other templated config is written, and it is RequiredBy=preconfigured.target, while kubelet.service waits on configured.target. So the device would still be online well before kubelet starts, in the same phase as the rest of the settings-derived configuration. Nothing in early boot needs swap — the non-swap variants have never had any at that point.

The default case still needs a decision

Deriving from thresholds does not by itself fix a stock node, because a stock node has T = 100 MiB absolute, which yields Z = 50 MiB — effectively disabling the feature. That is a real tension and I don't think it can be papered over: a meaningfully sized zram device is incompatible with kubelet's default 100 MiB absolute hard-eviction threshold.

So I'd suggest pairing the change with a default percentage threshold shipped alongside the swap feature, chosen so the invariant holds by construction — e.g. defaulting settings.kubernetes.eviction-hard to memory.available: 10% when release-swap is in the variant. Then Z = 5% · MemTotal satisfies Z ≤ 0.5·T on every node, at every size, with no user configuration.

If shipping a default threshold is unpalatable, the fallback is a plain RAM-proportional size — Z = min(5% · MemTotal, 4 GiB) — which is still strictly better than a constant, but leaves stock hard eviction unreachable.

For prior art, systemd's zram-generator defaults to min(RAM/2, 4096 MiB) and Fedora has sized zram proportionally since F33/F34. As far as I can tell, no other Kubernetes node OS (AL2023, Flatcar, Talos, COS) enables zram by default at all.

Option B — expose it as a setting

settings.swap.zram-size, accepting an absolute size or a percentage, with "auto" meaning the Option A derivation. More flexible, and now a smaller delta since Option A already introduces the template and the ordering change — but it needs a model type in sources/models and boot-only restart semantics. I'd suggest Option A as the default behaviour regardless, with B layered on later if there is demand.

One caveat to be explicit about

zramctl --size is uncompressed capacity. At the 3–4× ratios we measured, a device of size Z costs only Z/r physically — but it hides up to the full Z from workingSet. All the math above is driven by the uncompressed figure, which is the conservative and correct one for eviction reasoning; the physical cost of the proposed sizes is 3–4× smaller again.

Related

• #4075 — the de-facto thread for these symptoms; multiple reporters plus a maintainer confirm kswapd saturating CPU
• #4735 / core-kit#590, core-kit#792 — where zram was introduced and defaulted
• #4808 — requests swap files; does not cover zram sizing
• kubernetes#43916 — the page-cache half of the accounting blind spot