Kopia repo creation fails for every namespace but the first one created against a shared BackupStorageLocation ("cipher: message authentication failed")

What steps did you take and what happened:

1. Defined a BackupStorageLocation (any name/prefix works, doesn't need to be default).
2. Backed up one namespace with snapshotMoveData: true against that location, which was successful. This is the first-ever Kopia repository creation for that location.
3. Backed up a different namespace against the same BackupStorageLocation, which failed:

error to connect backup repo: error to connect repo with storage: error to connect to repository:
unable to create shared content manager: error loading indexes: error downloading indexes:
error loading index blob xn0_<hash>-c1: decrypt blob: error decrypting BLOB xn0_<hash>-c1:
unable to decrypt content: cipher: message authentication failed

Debug logs show the mover pod creating its own repo and failing to decrypt its own index blob within the same second, same process:

• No repository found, creating one (pkg/repository/ensurer.go)
• failed to open repository: ... decrypt blob: ... message authentication failed (~1s later, same pod)

It seems to me that this is order-dependent, not namespace-dependent. Whichever is backed up first against a freshly created BackupStorageLocation reliably succeeds, and the next one against that same location reliably fails, regardless of which two namespaces are used. I've reproduced this on two independent, fully clean setups (empty bucket, zero pre-existing BackupRepository/Backup/DataUpload objects, freshly restarted velero deployment and node-agent) with identical results both times.

Once a repo is corrupted this way it is permanent; every later backup/restore/maintenance attempt against it fails identically and does not self-heal.

At larger scale (full-cluster backup, ~35 namespaces, one shared BackupStorageLocation) this generalizes to the first namespace succeeding and everything after failing for the rest of the run, consistent with the two-step reproduction above.

What did you expect to happen:

Creating a second, independent Kopia repository under an existing BackupStorageLocation should not corrupt it. At minimum, a corrupt bootstrap should be self-detected and the repository recreated rather than left permanently broken.

Anything else you would like to add:

Based on the S3 logs, storage-layer integrity is confirmed to be intact, this is not corruption in transit or at rest. kopia.repository is written once and read back several times afterward, byte-count identical every time. The index blob is written once and read back once, byte-for-byte identical both ways. From what I can tell, the client received back exactly what it wrote, and still failed to decrypt it.

What I've been trying to rule out:

• Repository password / secret drift or caching: the password is mounted as a secret from an external vault
• Session/state accumulation in the long-running velero deployment or node-agent processes: I've restarted each independently across multiple test rounds; failure reproduces identically on fully fresh processes
• Accumulated Kubernetes object count (many prior Backup/DataUpload/BackupRepository objects): cleared these to zero, but failure still reproduces on the next attempt
• Generic S3 backend corruption: manual PutObject/GetObject round-trip of small test objects under the affected prefix were clean
• Namespace/app-level Kubernetes config (StorageClass, PVC size/mode, security context, annotations): identical across failing and succeeding namespaces, and irrelevant given the order-dependent finding above
• PR #10377 ("[1.18] Fix repo connection contest", included in v1.18.3-rc.1): tested with the minimal two-step reproduction above on a fully clean setup, with the failure reproducing identically

Given the order-dependence, this looks like state scoped to the BackupStorageLocation (or something keyed by it) inside Velero's repository-creation path being reused incorrectly for the second repository rather than freshly derived.

Environment:

• Velero version (use velero version): reproduced on both v1.18.1 and v1.18.3-rc.1
• Velero features (use velero client config get features): EnableCSI
• Kubernetes version (use kubectl version): v1.36.4
• Kubernetes installer & version: Talos Linux v1.13.9
• Cloud provider or hardware configuration: self-hosted S3-compatible object storage (MinIO), path-style addressing, TLS with a private CA
• OS (e.g. from /etc/os-release): Talos Linux (nodes); backup storage endpoint OS not applicable (S3-compatible appliance)

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here. Use the "reaction smiley face" up to the right of this comment to vote.

• for "I would like to see this bug fixed as soon as possible"
• for "There are more important bugs to focus on right now"