MicroVM hangs during SMP boot with ≥16 vCPUs on dual-socket NUMA hosts

When starting a microVM with a higher vCPU count (e.g., 16 vCPUs) on a dual-socket NUMA host, the guest occasionally hangs during SMP initialization.

During the hang, some Firecracker vCPU threads remain blocked in futex_wait_queue, and the guest kernel does not complete bringing up all secondary CPUs.

The issue occurs randomly during VM startup and has been observed when running Firecracker via jailer.

guest kernel version:5.4.116(The issue persists with 5.10.245 also ,when tried)

Architecture:             x86_64
CPU op-mode(s):         32-bit, 64-bit
Address sizes:          46 bits physical, 48 bits virtual
Byte Order:             Little Endian
CPU(s):                   32
On-line CPU(s) list:    0-31
Vendor ID:                GenuineIntel
Model name:             Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz
CPU family:           6
Model:                85
Thread(s) per core:   2
Core(s) per socket:   8
Socket(s):            2
Stepping:             4
BogoMIPS:             4200.00


Virtualization features:  
  Virtualization:         VT-x
Caches (sum of all):      
  L1d:                    512 KiB (16 instances)
  L1i:                    512 KiB (16 instances)
  L2:                     16 MiB (16 instances)
  L3:                     22 MiB (2 instances)
NUMA:                     
  NUMA node(s):           2
  NUMA node0 CPU(s):      0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
  NUMA node1 CPU(s):      1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31
Vulnerabilities:          
  Gather data sampling:   Mitigation; Microcode
  Itlb multihit:          KVM: Mitigation: Split huge pages
  L1tf:                   Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
  Mds:                    Mitigation; Clear CPU buffers; SMT vulnerable
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Mitigation; Clear CPU buffers; SMT vulnerable
  Reg file data sampling: Not affected
  Retbleed:               Mitigation; IBRS
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected;   BHI Not affected
  Srbds:                  Not affected
  Tsx async abort:        Mitigation; Clear CPU buffers; SMT vulnerable

Configure the microVM with 16 vCPUs.

Boot a Linux guest kernel.

Observe that VM startup occasionally hangs during SMP initialization.

The issue does not occur consistently, but appears randomly when starting the VM.

[    1.745873] smp: Bringing up secondary CPUs ...
[    1.746812] x86: Booting SMP configuration:

The boot process does not proceed further.

During the hang, Firecracker shows all vCPU threads created:

   PID    SPID TTY          TIME CMD
2358731 2358731 pts/4    00:00:01 firecracker
2358731 2358743 pts/4    00:00:01 fc_vcpu 0
2358731 2358744 pts/4    00:00:00 fc_vcpu 1
2358731 2358745 pts/4    00:00:00 fc_vcpu 2
2358731 2358746 pts/4    00:00:00 fc_vcpu 3
2358731 2358747 pts/4    00:00:00 fc_vcpu 4
2358731 2358748 pts/4    00:00:00 fc_vcpu 5
2358731 2358749 pts/4    00:00:00 fc_vcpu 6
2358731 2358750 pts/4    00:00:00 fc_vcpu 7
2358731 2358751 pts/4    00:00:00 fc_vcpu 8
2358731 2358752 pts/4    00:00:00 fc_vcpu 9
2358731 2358753 pts/4    00:00:00 fc_vcpu 10
2358731 2358754 pts/4    00:00:00 fc_vcpu 11
2358731 2358755 pts/4    00:00:00 fc_vcpu 12
2358731 2358756 pts/4    00:00:00 fc_vcpu 13
2358731 2358757 pts/4    00:00:00 fc_vcpu 14
2358731 2358758 pts/4    00:00:00 fc_vcpu 15

Inspecting a stuck vCPU thread shows it blocked in a futex wait

[<0>] futex_wait_queue+0x60/0x90
[<0>] futex_wait+0x185/0x270
[<0>] do_futex+0x106/0x1b0
[<0>] __x64_sys_futex+0x8e/0x1d0
[<0>] do_syscall_64+0x55/0xb0
[<0>] entry_SYSCALL_64_after_hwframe+0x6e/0xd8

The microVM should boot normally and the guest OS should complete SMP initialization and start executing the init process.