node-agent: data-path watcher goroutine outlives deletion of its own DataUpload CR, permanently holding a loadConcurrency slot

What steps did you take and what happened:

While diagnosing a separate DataUpload livelock (matching #9453's symptom shape — DataUploads stuck in Prepared with zero progress for 95+ minutes, zero bytes moved, no errors at INFO level), we took a live goroutine dump of a node-agent pod before restarting it (kubectl debug --target <pod> --image=busybox --target=node-agent -- sh -c 'kill -QUIT <pid>', since the distroless velero image ships no shell to send the signal directly).

The dump surfaced a second, distinct defect: a microServiceBRWatcher.startWatch goroutine (pkg/datapath/micro_service_watcher.go:259/271) still alive and actively logging for a DataUpload named chl-retrigger-velero-daily-full-1785844826-8srxb — a backup from 5 days earlier than the dump. Its DataUpload custom resource no longer exists (kubectl get datauploads.velero.io chl-retrigger-velero-daily-full-1785844826-8srxb -n velero → NotFound), yet the in-process watcher was still alive, still logging events, and still retrying a pod-kill that will never succeed:

time="2026-08-09T10:57:56Z" level=info msg="Received event for data path chl-retrigger-velero-daily-full-1785844826-8srxb, reason: FailedKillPod, message: error killing pod: [failed to \"KillContainer\" for \"2d6aafb0-0fc2-4efa-8c84-b26633c9022d\" with KillContainerError: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\", failed to \"KillPodSandbox\" for \"e1dbb0e8-d05a-44b5-ae8a-b187be83dd0a\" with KillPodSandboxError: \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\"]" controller=dataupload dataupload=velero/chl-retrigger-velero-daily-full-1785844826-8srxb logSource="pkg/datapath/micro_service_watcher.go:357"
time="2026-08-09T10:57:56Z" level=info msg="Received event for data path chl-retrigger-velero-daily-full-1785844826-8srxb, reason: Killing, message: Stopping container 2d6aafb0-0fc2-4efa-8c84-b26633c9022d" controller=dataupload dataupload=velero/chl-retrigger-velero-daily-full-1785844826-8srxb logSource="pkg/datapath/micro_service_watcher.go:357"

The same pair of log lines repeats on a roughly 4-minute cadence (observed again at 11:01:58/59Z, 11:06:01Z, 11:10:04Z — i.e. it has been retrying, unsuccessfully, since well before this observation window and shows no sign of ever stopping on its own).

The goroutine stack for this watcher, from the SIGQUIT dump:

goroutine 4519 gp=0xc00135f180 m=nil [sync.Cond.Wait, 1 minutes]:
...
github.com/vmware-tanzu/velero/pkg/datapath.(*microServiceBRWatcher).startWatch.func1()
	/go/src/github.com/vmware-tanzu/velero/pkg/datapath/micro_service_watcher.go:271 +0x18a
created by github.com/vmware-tanzu/velero/pkg/datapath.(*microServiceBRWatcher).startWatch in goroutine 246
	/go/src/github.com/vmware-tanzu/velero/pkg/datapath/micro_service_watcher.go:259 +0x65

(A second, independent instance of the same watcher for the same stale DataUpload name was also present elsewhere in the same dump, both blocked identically.)

What did you expect to happen:

A microServiceBRWatcher for a DataUpload whose CR has been deleted should eventually be torn down (or at minimum stop retrying indefinitely) rather than persisting for days, since — per our own investigation of a related but distinct issue (#9453) — node-agent's loadConcurrency limits the number of concurrent data-path operations per node. A watcher that never exits appears to permanently occupy one of those concurrency slots, which in our case left only 1 of 2 configured slots usable on the affected node for 5+ days, contributing to (though not fully explaining on its own) a subsequent full livelock of that node's DataUpload processing.

Environment:

• Velero version: v1.18.0 (docker.io/velero/velero:v1.18.0)
• Velero features: EnableCSI
• Kubernetes version: v1.33.11+rke2r1
• node-agent config: loadConcurrency.globalConfig: 2, prepareQueueLength: 8
• Cloud provider: self-hosted (Rook Ceph + rgw, S3-compatible offsite target)

Relationship to #9453:

We believe this is a distinct defect, not a duplicate: #9453 describes a live DataUpload CR whose reconciler stops receiving watch events and stalls in Prepared. This report describes a watcher goroutine that outlives the deletion of its own CR entirely — the reconcile loop and the CR are gone, but the async watcher spawned for it is not. A fix for #9453 (including Prepared-phase tasks in the periodic re-enqueue source) would not obviously address a goroutine that has no CR left to reconcile against. Happy to provide the full goroutine dump (16000+ lines) if useful — trimmed here to the relevant excerpt.