[Bug] vsock: 100% CPU main-loop livelock and hung connections after Resume with in-flight traffic (v1.16.x, EVENT_IDX)

Describe the bug

After a PATCH /vm {"state":"Paused"} → PATCH /vm {"state":"Resumed"} cycle, if there was in-flight bidirectional vsock traffic at pause time, the vsock device wedges on resume:

1. The Firecracker main/event-loop thread spins at 100% CPU in a tight epoll_pwait loop, and
2. All vsock connections stop making progress — existing connections hang and new host-initiated connections are never accepted by the guest.

The vCPUs run (guest is not paused) but block in KVM_RUN waiting for device interrupts that the wedged event loop never delivers, so the guest goes effectively dark (no serial output, no network).

This looks like the Pause/Resume-path analogue of #5969 / #5958, which fixed the equivalent hang only for the snapshot LoadSnapshot path. The Resume API path has no equivalent re-arm and additionally exhibits a busy-poll livelock. Both appear rooted in the vsock VIRTIO_RING_F_EVENT_IDX support added in #5872 (first shipped in v1.16.0).

To Reproduce

Minimal shape (does not require snapshots):

1. Boot a microVM with a vsock device and a guest agent listening on a vsock port.
2. Establish steady bidirectional vsock traffic — e.g. a guest process writing to stdout streamed host→guest over one connection (host→guest RX active), while the host periodically sends requests (guest→host TX active). The key is that both an RX descriptor and a TX descriptor are in-flight/unconsumed at the instant of pause.
3. PATCH /vm {"state":"Paused"}.
4. PATCH /vm {"state":"Resumed"}.
5. From the host, open a new vsock connection to the guest agent (e.g. an exec-style control connection).

Observed: step 5's CONNECT is never accepted; the host side blocks indefinitely. Firecracker's main thread is pinned at 100% CPU. Existing connections also stop advancing.

It is intermittent across cycles — it triggers on whichever pause happens to catch a descriptor in-flight, so a workload with continuous traffic (a 1 Hz stdout writer plus periodic control RPCs) reproduces it reliably within one or two pause/resume cycles.

Diagnostics captured on a wedged instance

Firecracker v1.16.1, x86_64, MMIO transport, two vCPUs. Captured live while the VM was hung.

Main thread — 100% CPU, tight epoll loop. strace -c -p <fc-main-tid> over 3 s:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.86    0.491726           1    330917           epoll_pwait
  0.14    0.000703         117         6           write
  0.00    0.000008           2         3           read

~110k epoll_pwait/s, each returning immediately. Per-event trace shows a nested-epoll cycle that never clears:

epoll_pwait(30, [{events=EPOLLIN, data={u32=13, u64=13}}], 32, 0, ...) = 1   # inner eventpoll
epoll_pwait(9,  [{events=EPOLLIN, data={u32=30, ...}}],     256, -1, ...) = 1  # outer -> inner
# repeats forever; fd 13 keeps reporting EPOLLIN and is never drained/deregistered

fd 13 is an accepted connection on the vsock backend Unix socket (the UDS the muxer accepts host-side connections on) — i.e. pending host→guest data on a muxed vsock connection that the device muxer never consumes and never removes from the epoll set → level-triggered busy-poll.

vCPU threads — idle, blocked in KVM_RUN.

fc_vcpu 0: State S (sleeping), blocked in ioctl(KVM_RUN), ~0 CPU growth over 10s
fc_vcpu 1: State S (sleeping), blocked in ioctl(KVM_RUN), ~0 CPU growth over 10s

Guest is waiting for device interrupts that the livelocked event loop never injects (ping to guest dead, serial silent after resume).

Firecracker warnings at every resume (guest console / firecracker log):

[vm:fc_vcpu 0:WARN] Received a VcpuEvent::Resume message with immediate_exit enabled. immediate_exit was disabled before proceeding
[vm:fc_vcpu 1:WARN] Received a VcpuEvent::Resume message with immediate_exit enabled. immediate_exit was disabled before proceeding
[vm:main:WARN] Got a spurious notification from api thread

Expected behaviour

After Resume, the vsock device should re-process any in-flight RX/TX descriptors and re-arm notifications (as the LoadSnapshot kick() path was fixed to do in #5958), so existing connections resume and new connections are accepted. The event loop must not busy-poll a level-triggered backend fd whose data cannot currently be delivered.

Hypothesis (for maintainers)

Two stacked effects, both downstream of vsock EVENT_IDX (#5872):

1. Notification state not re-armed across Resume. With VIRTIO_RING_F_EVENT_IDX negotiated, a descriptor in-flight at pause leaves the used/avail-event indices in a state where, after resume, the guest's TX kick and/or the host's RX interrupt is suppressed and never retried — the same class of bug #5958 fixed for restore, but the Resume API path has no equivalent replay/re-arm hook.
2. Muxer busy-poll under backpressure. When host→guest RX cannot be delivered (guest not consuming because of (1)), the muxer leaves the connection's backend UDS fd registered with level-triggered EPOLLIN and never drains or deregisters it, pinning the main loop at 100% CPU and starving all other device servicing — which is why the whole VM (net, serial) goes dark, not just vsock.

The restore fix (#5958) replays the TX queue kick in the device's restore kick(). Resume appears to need the analogous re-arm, plus (or including) correct edge/level handling of the muxer connection fd so a non-drainable connection doesn't livelock the loop.

Note on scope: effect (2), the busy-poll, may be at least partly independent of EVENT_IDX — a level-triggered backend fd that is never drained or deregistered is a muxer event-handling issue that EVENT_IDX exposes (by leaving RX undeliverable) rather than causes. If so, disabling vsock EVENT_IDX would hide the livelock but not fix the underlying fd-handling. This is a reason to fix the re-arm + muxer handling rather than treat it purely as an EVENT_IDX regression.

Environment

• Firecracker: v1.16.1 (release binary). Also expected on v1.16.0 (EVENT_IDX for vsock landed in #5872 → v1.16.0).
• Host arch: x86_64. vsock transport: MMIO. 2 vCPUs.
• Setup: Firecracker running inside a virtualized host (nested virtualization); the vsock device backs a containerd-shim (nerdbox) control channel to an in-guest agent. The bug is in Firecracker's vsock device event loop and is not specific to nesting.
• Guest kernel / rootfs: custom guest kernel + minimal initrd (details available on request; not believed relevant — the guest driver is stock virtio-vsock).
• Not using snapshots for this repro — plain PauseVM/ResumeVM API.

Additional context

• Related: #5872 (adds VIRTIO_RING_F_EVENT_IDX to vsock, v1.16.0), #5958 / #5969 (restore-path vsock hang, fixed for LoadSnapshot only in v1.16.1).
• The 100% CPU main-loop symptom (distinct from the plain connection hang in #5969) may be the more actionable signal, since it's a deterministic busy-poll on a specific fd rather than a race — happy to capture additional traces (full per-event epoll_pwait dump, muxer state, queue indices) or provide a self-contained repro harness on request.