Baike.dev
Log in
Back to tool/Back to issues
#41622·WSL

rtnl deadlock after hv_netvsc sub-channel ring allocation fails (ENOMEM) when a VPN adapter is hot-added in mirrored networking - follow-up to #41474

Author: zaknaCreated Sep 16, 2026Updated Sep 17, 2026
Labelskernel

logs.etl.txt

logs.etl.txt

Windows Version

10.0.26200.9168

WSL Version

2.7.13.0

Are you using WSL 1 or WSL 2?

  • WSL 2
  • WSL 1

Kernel Version

6.18.33.2-2 (uname -r: 6.18.33.2-microsoft-standard-WSL2)

Distro Version

Ubuntu 24.04.4 LTS

Other Software

  • Cisco AnyConnect / Secure Client VPN (adapter "Cisco AnyConnect Virtual Miniport Adapter for Windows x64").
  • .wslconfig: memory=10GB, swap=4GB, processors=11, networkingMode=mirrored, dnsTunneling=true, autoProxy=false, [experimental] autoMemoryReclaim=gradual, sparseVhd=true.
  • Normal developer load inside the VM (Node.js tooling, several SSH sessions).

Repro Steps

Same kernel failure as #41474, which was closed because only post-recovery logs were attached. This report is captured while the VM is still deadlocked, and it has a different, common trigger: no small memory cap, just a VPN connecting on the Windows host.

  1. WSL2 running for about 1.5 days with networkingMode=mirrored and autoMemoryReclaim=gradual, 10 GB memory limit.
  2. On Windows, connect Cisco AnyConnect. Mirrored networking hot-adds the VPN adapter into the VM as eth1.
  3. vmbus_add_channel_work on the hv_sub_chan workqueue tries to allocate the sub-channel ring buffer (order 8, about 1 MiB contiguous, GFP_KERNEL|__GFP_ZERO) and fails, although about 6.5 GB is available: memory is fragmented (see buddyinfo).
  4. netvsc_sc_open logs sub channel open failed: -12 seven times.
  5. From that moment every rtnetlink dump blocks forever in rtnl_dumpit, in state D. Anything that calls getifaddrs() or lists interfaces hangs and cannot be killed: ip addr, ss, sshd connection children (before the SSH banner), npm / Node.js, sudo. Load average climbs with each new stuck process (41 at capture time, 36 processes in D state).

It is not reproducible on demand, because it needs fragmented kernel memory at the moment the adapter is hot-added. The host had 40 earlier order-7 ring allocation failures on the hv_pri_chan workqueue (vmbus_probe / vmbus_device_register path) over the previous day, and none of them hung anything. The single failure on the sub-channel path (netvsc_sc_open) is the one that left rtnl blocked.

Expected Behavior

A failed sub-channel open should unwind and release rtnl, so the interface keeps working with fewer channels, or is removed cleanly. Netlink queries should never block permanently.

Actual Behavior

rtnl stays blocked indefinitely. The only recovery is wsl --shutdown. All networking-related process creation in the distro is wedged (SSH server unusable, Node.js tools hang at startup, sudo hangs).

Allocation failure (sanitized; dmesg -T wall-clock times drift in WSL, the order of events is reliable):

kworker/0:1: page allocation failure: order:8, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 0 UID: 0 PID: 18457 Comm: kworker/0:1 Not tainted 6.18.33.2-microsoft-standard-WSL2 #1 PREEMPT(none)
Workqueue: hv_sub_chan vmbus_add_channel_work
Call Trace:
 <TASK>
 dump_stack_lvl+0x70/0x90
 dump_stack+0x14/0x20
 warn_alloc+0x10a/0x180
 ? __alloc_pages_direct_compact+0x1a6/0x250
 __alloc_pages_slowpath.constprop.0+0x2a3/0xe30
 __alloc_frozen_pages_noprof+0x31c/0x330
 __alloc_pages_noprof+0x12/0x30
 vmbus_alloc_ring+0x78/0xd0
 ? __pfx_netvsc_channel_cb+0x10/0x10
 vmbus_open+0x29/0x80
 ? rtnl_is_locked+0x19/0x30
 netvsc_sc_open+0x11a/0x1a0
 vmbus_add_channel_work+0x101/0x1d0
 process_one_work+0x19b/0x3c0
 worker_thread+0x1a3/0x320
 kthread+0x111/0x240
 ret_from_fork+0x13c/0x1a0
 ret_from_fork_asm+0x1a/0x30
 </TASK>
...
hv_netvsc <device-guid> eth1: sub channel open failed: -12   (x7)

Every blocked process has the same stack (read as root from /proc/<pid>/stack):

=== comm ip state D (disk sleep)
[<0>] rtnl_dumpit+0x7f/0xb0
[<0>] netlink_dump+0x188/0x3c0
[<0>] __netlink_dump_start+0x1fc/0x330
[<0>] rtnetlink_rcv_msg+0x2a6/0x3f0
[<0>] netlink_rcv_skb+0x60/0x110
[<0>] rtnetlink_rcv+0x19/0x20
[<0>] netlink_unicast+0x282/0x3d0
[<0>] netlink_sendmsg+0x20c/0x440
[<0>] __sys_sendto+0x1ee/0x200
[<0>] __x64_sys_sendto+0x28/0x30
[<0>] x64_sys_call+0x15e7/0x20f0
[<0>] do_syscall_64+0x73/0x990
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

Fragmentation at capture time (/proc/buddyinfo, orders 0-10), MemAvailable 6,533,488 kB of 10,183,560 kB:

Node 0, zone    DMA32  11829   7332   5518   4144   2494   1426    866    463    172     39      4
Node 0, zone   Normal   6096   3637   3034   1638    914    182    137     61     29     35      8

A possible code path, for the maintainers to confirm: the sub-channel setup in netvsc waits under rtnl for all requested sub-channels to report open (open_chn == num_chn). If one netvsc_sc_open fails with ENOMEM, that count may never be reached, so the waiter keeps rtnl held. We could not see the holder directly: the distro PID namespace hides kernel threads, and the echo w > /proc/sysrq-trigger dump in the kernel log was truncated after 9 of 36 blocked tasks.

Diagnostic Logs

I cannot run collect-wsl-logs.ps1: this is a managed corporate laptop without Windows administrator rights, so no logs.etl can be produced. Please do not auto-close for the missing ETL; everything below was collected during the deadlock from inside the VM (root through wsl -u root), plus non-admin Windows output:

  • dmesg-subchannel-failure.txt - the failing allocation, Mem-Info and buddy lists at failure time, and the sub channel open failed lines
  • dmesg-primary-channel-failure-example.txt - one of the 40 earlier order-7 hv_pri_chan failures that did not hang
  • waiter-stacks.txt - kernel stacks of blocked ip, sshd and npm processes
  • dstate-processes.txt - all 36 blocked processes with their wait channels
  • sysrq-w-blocked-tasks.txt - echo w > /proc/sysrq-trigger output (truncated by the kernel log)
  • meminfo.txt, buddyinfo.txt, wsl-version.txt, wslconfig.txt, windows-adapters.txt

Device GUIDs, MAC and IP addresses and user names are replaced by placeholders. The full kernel log is available on request. The VM had to be restarted with wsl --shutdown after this capture, because every networking-related process creation was wedged. If it happens again, tell me which extra kernel-side capture you want taken before the restart.

Related: #41474 (same failure with a low memory= cap, closed for missing logs).

dmesg-subchannel-failure.txt dmesg-primary-channel-failure-example.txt waiter-stacks.txt dstate-processes.txt sysrq-w-blocked-tasks.txt meminfo.txt buddyinfo.txt wsl-version.txt wslconfig.txt windows-adapters.txt

Source: microsoft/WSL

View original on GitHubView discussion on GitHub