[Kernel Proposal] Prevent VMBus Control-Plane Starvation & High-Order Ring Allocation Deadlocks Under Memory Pressure

Problem Statement & Architectural Context

Under heavy memory allocation (dense container compilation, heavy memory tiering, or active swap pressure), Microsoft WSL2 instances frequently encounter severe control-plane timeouts:

1. HCS Watchdog Timeout: When guest available memory drops into direct reclaim, atomic page allocations (GFP_ATOMIC) required by synthetic VMBus packet rings and host heartbeat channels fail. The Windows Host Compute System infers a hard lock and terminates the instance with Wsl/Service/E_UNEXPECTED (0x8000ffff) or drops the virtual switch (Hyper-V-VmSwitch Event 102/291).
2. Order-7 Allocation Deadlock: Hyper-V synthetic channel rings (vmbus_alloc_ring()) require contiguous physical 512 KiB allocations (order:7, mode:0xdc0(GFP_KERNEL|__GFP_ZERO)). Under normal session fragmentation, Order-7 is completely exhausted even when gigabytes of physical memory remain free across orders 0–3, completely wedging session creation.

Root Cause Forensic Evidence

Buddy Allocator State at Failure (/proc/buddyinfo):

Node 0, zone Normal 815 420 120 40 12 8 3 0 0 0 0

(Orders 7–10 are 0; allocator cannot service 512 KiB contiguous blocks).

Kernel Allocation Trace:

kswapd0: page allocation failure: order:0, mode:0x800(GFP_ATOMIC)
Call Trace:
  dump_stack_lvl+0x48/0x70
  netvsc_alloc_recv_comp+0x28/0x60 [hv_netvsc]
  vmbus_onoffer+0x110/0x240 [hv_vmbus]
hv_balloon: balloon inflation requested: 131072 pages (512 MB) from host
hv_balloon: page allocation failure in alloc_balloon_pages (competing with direct reclaim)

The Proposed Fixes (Validated on Kernel 6.18.40.1)

1. Dynamic VMBus Atomic Headroom & Balloon Backpressure (drivers/hv/hv_common.c, drivers/hv/hv_balloon.c)

• Calibrates vm.min_free_kbytes dynamically during late_initcall (clamped up to 512 MiB) to ensure dedicated reservation for synthetic VMBus channels.
• Defers host balloon inflation (hv_balloon) when guest memory is constrained (si_mem_available() < totalram_pages() / 32), preventing balloon thrashing during direct reclaim.

2. High-Order Virtual Memory Fallback (drivers/hv/ring_buffer.c, drivers/hv/channel.c)

• Falls back to vzalloc() when physical contiguous order-7 allocation fails.
• Translates virtually allocated pages into PFNs for the GPA descriptor table via vmalloc_to_page(), providing 100% transparent operation to the Windows Hyper-V host.
• Fully preserves Confidential VM (Azure CVM / AMD SEV-SNP) memory encryption lifecycle checks (vfree() on decrypted pages).

Empirical Hardware Validation

Tested on physical hardware under WSL2 2.7.14.0 (Kernel 6.18.40.1-microsoft-standard-WSL2+, NVIDIA GeForce RTX 2060):

• 99% RAM Pressure (14.7 GB allocation): Sustained 100% hold with PASS_ZERO_PANIC (Stock WSL2 times out within 45s).
• Synthetic Channel Latency Under Zero Order-7 Blocks: Established cleanly in ≤ 0.15 ms via virtual ring buffer.
• Teardown & Recovery: 10+ GB clean restored RAM to host upon workload completion.

Reference Implementation

Complete reference code and build instructions are live on: emersonbusson/WSL2-Linux-Kernel