virt-handler ghost record reconciliation fails after downgrade due to socket path normalization mismatch (/pods/... vs //pods/...)
What happened:
When downgrading KubeVirt with running VMIs still present, virt-handler can fail to reconcile an existing ghost record for a running VMI because the stored socket path and the rediscovered live socket path differ only by path normalization.
In the failing case, the ghost record contains a socket path like:
/pods/<podUID>/volumes/kubernetes.io~empty-dir/sockets/launcher-sock
but the downgraded virt-handler rediscovers the live socket as:
//pods/<podUID>/volumes/kubernetes.io~empty-dir/sockets/launcher-sock
These refer to the same location semantically, but AddGhostRecord() compares them as raw strings and rejects the record with:
can not add ghost record when entry already exists with differing socket file location
After that, the VMI is repeatedly re-enqueued and is not managed normally.
This appears to be caused by a change where newer KubeVirt versions normalize the socket path with filepath.Clean(), while older versions do not.
What you expected to happen:virt-handler should not fail ghost record reconciliation just because one path is stored as /pods/... and the other is discovered as //pods/....
Even if downgrade with running VMIs is not considered a supported operational scenario, logically equivalent socket paths should be normalized or canonicalized before comparison so that reconciliation does not fail for the same underlying socket location.
How to reproduce it (as minimally and precisely as possible):
- Start with a KubeVirt version where the socket path helper uses
filepath.Clean(). - Start one or more VMIs and let them remain running.
- Downgrade KubeVirt to a version where the socket path helper does not use
filepath.Clean(). - Let the downgraded
virt-handlerrediscover the running VMI and its launcher socket. - Observe that ghost record reconciliation fails with:
can not add ghost record when entry already exists with differing socket file location
A successful comparison case can also be seen when both the writing version and the reading version use the same unnormalized //pods/... path format. In that case, reconciliation succeeds.
Additional context:
The issue appears to come from inconsistent socket path canonicalization across versions.
Example of the relevant code pattern:
Newer version:
func SocketDirectoryOnHost(podUID string) string {
return filepath.Clean(fmt.Sprintf("/%s/%s/volumes/kubernetes.io~empty-dir/sockets", podsBaseDir, podUID))
}
Older version:
func SocketDirectoryOnHost(podUID string) string {
return fmt.Sprintf("/%s/%s/volumes/kubernetes.io~empty-dir/sockets", podsBaseDir, podUID)
}
With podsBaseDir = "/pods", the formatted path becomes "//pods/..." before cleaning. So one version stores /pods/... while the other later rediscovers //pods/....
The issue is therefore not that the socket location is actually different, but that ghost record reconciliation depends on raw string equality for a path that is not normalized consistently across versions.
It looks like AddGhostRecord() should normalize/canonicalize both paths before comparing them.
Environment:
KubeVirt version (use virtctl version): observed on downgrade from v1.7.0 to v1.5.0
Kubernetes version (use kubectl version): N/A
VM or VMI specifications: VMI was already running before downgrade
Cloud provider or hardware configuration: N/A
OS (e.g. from /etc/os-release): N/A
Kernel (e.g. uname -a): N/A
Install tools: N/A
Others: downgrade from v1.5.0 to v1.1.0 did not reproduce the issue because both versions used the same unnormalized //pods/... path format
Source: kubevirt/kubevirt