VMI masquerade interface reports guest IP (10.0.2.2) instead of pod IP once guest-agent connects — only for GPU/host-device VMs with many guest interfaces (regression since ~v1.8)
What happened
On KubeVirt v1.9.0, a VM using masquerade binding on the pod network plus a PCI host-device (GPU) passthrough, running qemu-guest-agent, reports its primary interface's status.interfaces[0].ipAddress as 10.0.2.2 (the masquerade guest-side gateway address) instead of the virt-launcher pod IP.
The value flips during startup of a single VM:
- Before guest-agent connects:
status.interfaces[0].ipAddress = <pod IP>(correct;infoSource: domain). - After
AgentConnected=True: it becomes10.0.2.2and stays there permanently (infoSource: domain, guest-agent).
# before agent: phase=Running ip0=10.244.1.120 agent=
# after agent: phase=Running ip0=10.0.2.2 agent=True <-- flips and sticksWhat you expected
status.interfaces[].ipAddress for the masquerade interface should report the virt-launcher pod IP (as it does for non-GPU VMs and before guest-agent connects), regardless of guest-agent state.
Impact
Any consumer that reads the VMI's reported interface IP to reach the VM breaks. Concretely, cluster-api-provider-kubevirt (CAPK)'s default ssh bootstrap check dials the reported interface IP to read the CAPI sentinel file; with 10.0.2.2 (not routable from the pod network) it never connects, so the KubevirtMachine is stuck BootstrapExecSucceeded=False and the CAPI Machine never leaves Provisioning — even though the guest booted fine. (Workaround for CAPK users: set virtualMachineBootstrapCheck.checkStrategy: none.)
Scoping (all on the same cluster, KubeVirt v1.9.0)
| VM | masquerade | guest-agent | GPU passthrough | many guest ifaces (in-guest CNI) | reported default IP |
|---|---|---|---|---|---|
| K8s worker, no GPU | ✔ | ✔ | ✘ | ✔ (Cilium) | pod IP ✅ |
| Standalone VM, GPU | ✔ | ✔ | ✔ | ✘ (single iface) | pod IP ✅ |
| K8s worker, GPU | ✔ | ✔ | ✔ | ✔ (Cilium) | 10.0.2.2 ❌ |
→ Neither GPU passthrough alone nor a many-interface guest alone reproduces it; only the combination does. The broken VM's masquerade interface is status.interfaces[0] (name default), infoSource: domain, guest-agent, ipAddress: 10.0.2.2, while AgentConnected=True.
Steps to reproduce
- KubeVirt v1.9.0, a VM with
masqueradebinding on the pod network andqemu-guest-agentrunning in the guest. - Attach a PCI host device (e.g. an NVIDIA GPU via
permittedHostDevices+spec.domain.devices.gpus). - Have the guest bring up many interfaces (e.g. the VM is a Kubernetes node running Cilium, so guest-agent reports >10 interfaces).
- Wait for
AgentConnected=True. kubectl get vmi <name> -o jsonpath='{.status.interfaces[0].ipAddress}'→10.0.2.2instead of the pod IP.
Environment
- KubeVirt: v1.9.0
- Kubernetes: v1.36.3
- Binding:
masquerade(pod network, noportslist) - Guest: Ubuntu 24.04,
qemu-guest-agentrunning; Cilium CNI (VM is a CAPK-provisioned worker node) - Host device: NVIDIA L40S PCI passthrough via
permittedHostDevices(10DE:26B9) - CAPK (consumer): v0.10.5
Additional context / possible origin
- The same setup worked on KubeVirt v1.6.0; it broke after upgrading to v1.9.0.
- Possibly related to the v1.8.0 VMI interface-status reporting rework — pointers for triage:
- #16391 "Limit the number of guest-only interfaces reported on the VMI status to 10"
- #16242 "Omit LLA from the status report when using masquerade binding"
- #17145 "Fixed an infinite VMI status update loop… corrects interface ordering"
- Hypothesis (unconfirmed): the GPU host device shifts the guest's PCI/interface enumeration ordering; under the reworked status merge + the 10-interface cap, the pod-IP association for the masquerade interface is dropped and the guest-agent value (
10.0.2.2) wins. Maintainers can likely pinpoint from the interface-merge path.
The VM is healthy — only the reported IP is wrong
SSH to the VM on its pod IP succeeds (masquerade NAT works); inside the guest,
enp1s0 genuinely holds 10.0.2.2. The CAPI bootstrap sentinel
/run/cluster-api/bootstrap-success.complete is present (success) and kubelet
is active — the guest fully bootstrapped and joined the cluster. So the only
defect is status.interfaces[].ipAddress reporting 10.0.2.2 instead of the
(reachable) pod IP; a consumer that used the pod IP would succeed. This also
means CAPK's checkStrategy: none workaround is safe (it doesn't mask a real
bootstrap failure — the sentinel exists).
Source: kubevirt/kubevirt