runtime: physical VF restore races QEMU exit during normal sandbox teardown

Summary

During normal shutdown of a QEMU-backed Kata sandbox with a cold-plugged physical SR-IOV VF, the Go runtime can begin restoring the device from vfio-pci to its host driver before QEMU has exited and released its VFIO file descriptors.

The restore then stops while unbinding the device from vfio-pci. Kata never reaches drivers_probe, leaving the VF unbound with driver_override=mlx5_core. Subsequent workloads cannot use the VF without manual recovery.

Regression identified

Revert testing identifies d3291b877 (runtime: publish TaskExit before sandbox teardown) as the change that triggers this failure. With this commit present, normal pod teardown can leave the physical VF unbound. Reverting this commit fixes the issue and restores the VF successfully in end-to-end testing.

The commit publishes TaskExit before sandbox teardown and moves teardown outside s.mu. That ordering change appears to expose the QEMU/VFIO lifetime race described below.

Environment

• Kata Go runtime with QEMU
• Reproduced at effc0cdcebe36ad1c83c43b28ba4c5449c55f3a0
• Mellanox SR-IOV VF
• Host driver: mlx5_core
• Passthrough driver: vfio-pci
• QEMU: /usr/libexec/qemu-kvm
• Physical endpoint cold-plugged through a QEMU root port
• OpenShift/RHCOS host

Reproduction

1. Load vfio-pci on the host.
2. Start a Kata pod with a physical Mellanox SR-IOV VF cold-plugged through a QEMU root port.
3. Stop/delete the pod normally.
4. Repeat start/stop cycles and inspect each managed VF's driver and driver_override.

At effc0cdce, multiple VFs were left unbound, for example:

broken 0000:b5:09.2, driver UNBOUND
broken 0000:b5:0a.6, driver UNBOUND
broken 0000:b5:0b.5, driver UNBOUND

Observed behavior

For one VF, attachment completed normally:

01:49:56.084557 Write vfio-pci to driver_override
01:49:56.084678 Unbinding device from driver
01:49:57.596512 Writing bdf to drivers-probe-path
01:49:57.623840 Launching QEMU with vfio-pci,host=0000:b5:09.7

During normal teardown:

01:52:05.087939 Stopping VM
01:52:05.087959 QEMU: Stopping Sandbox
01:52:05.088359 QMP event channel closed
01:52:05.088412 Detaching physical endpoint
01:52:05.088638 Write mlx5_core to driver_override
01:52:05.089072 Unbinding device from driver

The kernel simultaneously reported:

vfio-pci 0000:b5:09.7: Relaying device request to user (#0)

There was no teardown-side Writing bdf to drivers-probe-path log. The final state was:

driver:          UNBOUND
driver_override: mlx5_core

Expected behavior

Normal sandbox teardown should not restore a physical VF until QEMU has exited and released the VFIO device. The restore should reach /sys/bus/pci/drivers_probe, and the VF should be bound back to mlx5_core.

Analysis

The relevant shutdown path is:

Sandbox.Stop
  -> stopVM
    -> qemu.StopVM
  -> removeNetwork
    -> LinuxNetwork.RemoveEndpoints
      -> PhysicalEndpoint.Detach
        -> BindDevicetoHost

At effc0cdce:

• qemu.StopVM(waitOnly=false) sends SIGKILL to QEMU and returns without waiting for the QEMU PID to exit.
• QEMU is reaped independently by LogAndWait() through qemuCmd.Wait().
• QMP channel closure and Kata's q.stopped flag do not establish that the QEMU process has exited or released its VFIO file descriptors.
• BindDevicetoHost() writes the original driver override, unbinds from vfio-pci, and only then writes the BDF to drivers_probe.
• The logs enter the VFIO unbind but never reach the following drivers_probe operation. The kernel's "Relaying device request to user" message is consistent with userspace still owning the VFIO device.

This points to a QEMU lifetime race between StopVM() and physical endpoint restoration.

Version comparison

A downstream kata-containers-3.31.0-3.rhaos4.22.el9 build restored the VFs successfully in 10 of 10 start/stop cycles.

I inspected that build's public SRPM:

• Its source is upstream 3.31.0 commit cec98e0d976bbf4cae016298ffea269f57294264.
• Its only downstream runtime source patch changes the default log level from Warn to Info.
• The relevant StopVM and LogAndWait implementation is identical to effc0cdce.
• It does not contain a downstream QEMU-exit wait before network teardown.

The successful downstream result therefore appears timing-dependent rather than evidence of a downstream source fix.

Related

• #13790 — rollback for physical VFIO attach failures; the failure reported here occurs during normal teardown and reproduces at the PR's parent commit.