Issue with incremental backup behavior with Velero 1.18 using CSI Snapshot Data Movement with the built-in Kopia data mover.

What steps did you take and what happened:

I am testing incremental backup behavior with Velero 1.18 + CSI Snapshot Data Movement + built-in Kopia data mover using an NFS CSI volume.

Velero version (use velero version): 1.18 Velero features (use velero client config get features): EnableCSI / CSI Snapshot Data Movement Kubernetes version (use kubectl version): v1.35.x Kubernetes installer & version: OpenStack HPOS Kubernetes Cloud provider or hardware configuration: OpenStack HPOS; NFS-backed storage CSI driver: nfs.csi.k8s.io StorageClass: nfs-client VolumeSnapshotClass: nfs-snapshot-class Backup Storage Location: default Data mover: Velero built-in data mover / Kopia OS: Linux (data mover node)

I created an NGINX application with a PVC and performed the following tests:

Backup 1: Initial volume contained ~1 GiB.

TOTAL BYTES:       1073741824
INCREMENTAL BYTES: 1073741824
Duration:           ~58 sec

Backup 2: Added another 1 GiB.

TOTAL BYTES:       2147483648
INCREMENTAL BYTES: 2147483648
Duration:           1m19.351s

Backup 5: No changes were made to the volume after the previous backup.

time velero backup create nginx-backup-5 \
  --include-namespaces velero-incremental-poc \
  --snapshot-move-data=true \
  --wait

Result:

TOTAL BYTES:       2252341248
INCREMENTAL BYTES: 2252341248
Duration:           1m20.974s

The DataUpload itself confirms the same:

Incremental Bytes: 2252341248

Progress:
  Bytes Done:  2252341248
  Total Bytes: 2252341248

Snapshot Type: CSI
Driver: nfs.csi.k8s.io

DataUpload event:

{
  "totalBytes": 2252341248,
  "incrementalBytes": 2252341248
}

So, even with no changes to the source PVC, the entire 2.25 GB volume is reported as incremental data.

What did you expect to happen:

For a backup with no changes since the previous backup, I expected:

TOTAL BYTES:       ~2252341248
INCREMENTAL BYTES: ~0

If only a small amount of data is added, I would expect INCREMENTAL BYTES to represent approximately the changed data.

The following information will help us better understand what's going on:

I can provide the Velero debug bundle and the following outputs:

velero version
velero client config get features
kubectl version

velero backup describe nginx-backup-5
velero backup logs nginx-backup-5

kubectl -n velero describe dataupload nginx-backup-5-8xqxz

The Kopia BackupRepository is healthy and has remained the same throughout the tests:

Name:             velero-incremental-poc-default-kopia
Repository Type:  kopia
Backup Location:  default
Volume Namespace: velero-incremental-poc
Phase:             Ready
Events:            <none>

Anything else you would like to add:

The main question is whether this behavior is expected with NFS CSI (nfs.csi.k8s.io), or whether there is a limitation/issue with incremental detection in the CSI Snapshot Data Movement/Kopia data mover path.

I understand that INCREMENTAL BYTES may not equal the actual physical storage written to the object store because Kopia can perform deduplication/compression. However, the DataUpload itself is reporting the entire filesystem as incremental, even when there are no changes.

Could you please confirm whether this is expected behavior or a known limitation with NFS CSI snapshots?