Kubelet OOM score calculation collapses to 999 for small/medium containers on large-memory nodes and ignores per-container memory guarantees

/sig node /kind bug

This bug description was assisted by AI.

What happened / What would you like to be added?

Update GetContainerOOMScoreAdjust in pkg/kubelet/qos/policy.go for Burstable QoS containers (guarded behind a new Kubelet feature gate) to fix precision collapse on large-memory nodes and recognize per-container memory guarantees:

1. Replace linear node-capacity division with logarithmic scaling (log2(1 + R) / log2(1 + C)) so container memory requests (R) across multiple orders of magnitude maintain distinct oom_score_adj values on large nodes (1 TiB to 4 TiB+).
2. Incorporate per-container memory guarantee ratio (R / L) so "memory-guaranteed" containers (memory.requests == memory.limits, such as orchestrators or control agents) receive lower OOM score adjustments than containers configured to overcommit/burst far above their requests (R << L or unlimited).
3. Reserve a safety gap between maximum Burstable baseline (800) and BestEffort (1000) so mild bursts do not immediately invert kill priority against BestEffort workloads.
4. Gate behind a Feature Gate so operators can roll out or revert to the legacy calculation without risk.

Why is this needed?

Currently, GetContainerOOMScoreAdjust computes oom_score_adj for Burstable containers as:

oom_score_adj = 1000 - (1000 * memoryRequest) / memoryCapacity

clamped to [3, 999], while the Linux kernel calculates process badness points as:

points = oom_score_adj + (usage_bytes * 1000) / memoryCapacity

When this heuristic was written in 2015, nodes typically had 4 GiB–64 GiB of RAM. On modern 1 TiB–4 TiB nodes running microservices, this formula breaks down in three ways:

1. Integer Quantization Collapse ("Everything is 999")

Because the formula divides by memoryCapacity in integer arithmetic across 1000 buckets, any container requesting less than memoryCapacity / 1000 evaluates to:

(1000 * memoryRequest) / memoryCapacity == 0  =>  oom_score_adj = 999

• On a 1 TiB (1024 GiB) node, every container requesting < 1.024 GiB gets oom_score_adj = 999. A container requesting 50 MiB, 256 MiB, or 1 GiB receives the exact same score (999).
• On a 4 TiB node, every container requesting < 4.096 GiB gets oom_score_adj = 999.
• Note: While #111356 observed this symptom and PR #128029 mitigated native sidecars by copying the minimum score of regular containers, regular containers themselves still collapse to 999 on large nodes (making the sidecar adjustment a no-op as well).

2. Loss of Per-Container Memory Guarantee (R / L)

Because Kubernetes QoS is evaluated at the Pod level across both CPU and memory, a container that explicitly sets memory.requests == memory.limits (zero memory overcommit) is forced into Burstable QoS whenever:

• The Pod has another container that is burstable (e.g., an Orchestrator + Executor Pod), or
• CPU limits are unset or differ from CPU requests (previously noted in #78848).

Consider a Burstable Pod on a 1 TiB node with two containers:

• Orchestrator (O): request = 500 MiB, limit = 500 MiB (R / L = 1.0, memory-guaranteed). Steady usage = 200 MiB.
• Executor (E): request = 500 MiB, limit = 32 GiB (R / L = 0.015, burst-heavy). Spikes to 1.5 GiB (3x request).

Under the current formula:

• oom_score_adj(O) = 999 => points(O) = 999 + (0.2 / 1024) * 1000 = 999.2
• oom_score_adj(E) = 999 => points(E) = 999 + (1.5 / 1024) * 1000 = 1000.5

The kernel badness score between a memory-guaranteed orchestrator using 40% of its request and an executor bursting 300% over its request differs by only 1 point. If the executor spawns multiple worker processes, each worker process has fewer kernel points than the orchestrator, causing the kernel OOM killer to kill the orchestrator first.

3. 1-Point Margin Between Burstable and BestEffort

Because small Burstable containers get oom_score_adj = 999 and BestEffort containers get 1000, the baseline margin is only 1 point (0.1% of host RAM). On a 1 TiB node, a Burstable container exceeding its request by 1.1 GiB reaches score 1000.1, making it more likely to be killed than an unreserved BestEffort container.

Proposed Formula

Replace the linear formula for Burstable containers with a two-factor formula combining per-container memory guarantee ratio (R / L) and logarithmic request sizing (log2(1 + R) / log2(1 + C)), anchored at a baseline of 800:

oom_score_adj = 800 - 400 * (R / L) - 350 * (log2(1 + R) / log2(1 + C))

Where:

• R = effective container memory request (including pod-level / DRA share if enabled)
• C = node memory capacity
• L = effective container memory limit (if unset or > C, L = C)
• Result is clamped to [3, 999] (with sidecar clamping preserved).

In Go, integer floor(log2(x)) + 1 can be computed in a single instruction via math/bits.Len64(uint64(x)) without floating-point arithmetic.

Concrete Comparison Across Node Sizes

Scenario 1: Large Node (C = 1 TiB = 1024 GiB, bits = 40)

Scenario 2: Small Node (C = 16 GiB, bits = 34)

Rollout & Why a KEP Is Not Required

• Feature Gate Protection: The updated calculation in GetContainerOOMScoreAdjust will be guarded behind a Kubelet feature gate, allowing safe rollout and immediate fallback to the legacy formula if needed.
• No Semantic or API Changes: This change does not alter any Kubernetes API, CRI protocol, or documented QoS contract:
  • The strict ordering Guaranteed (-997) < Burstable ([3, 999]) < BestEffort (1000) is preserved.
  • Within Burstable, the documented intent—that containers reserving more memory and staying within their bounds are protected over containers consuming memory beyond their requests—is unchanged and preserved on small nodes, while fixing the integer truncation bug where containers collapse to 999 on large nodes.
  • Prior refinements to GetContainerOOMScoreAdjust (such as PR #128029 for sidecars) followed the same path without requiring a standalone KEP.