Bug: DataUpload tasks can get permanently stuck in the `Prepared` phase
What happened?
While reviewing the DataUploadReconciler in data_upload_controller.go, I noticed that DataUpload tasks can get permanently stuck in the Prepared phase if they fail to progress for any reason (e.g., missing an update event or temporary delays).
This happens because the GenericEventPredicate used for the PeriodicalEnqueueSource in SetupWithManager completely omits the DataUploadPhasePrepared state:
gp := kube.NewGenericEventPredicate(func(object client.Object) bool {
du := object.(*velerov2alpha1api.DataUpload)
if du.Status.Phase == velerov2alpha1api.DataUploadPhaseAccepted {
return true
}
// ... missing check for DataUploadPhasePreparedBecause Prepared is missing from this predicate, the controller never periodically re-queues and re-checks tasks in this phase. If a task reaches Prepared but the subsequent watch event is dropped or delayed, it will just sit there indefinitely without ever timing out or progressing.
What did you expect to happen?
The controller should periodically re-enqueue DataUpload objects that are in the Prepared phase, just like it does for those in the Accepted phase. This ensures that the controller can recover from missed events or enforce timeouts if the preparation takes too long.
Proposed Fix
In pkg/controller/data_upload_controller.go, update the predicate in SetupWithManager to also return true when the phase is Prepared:
if du.Status.Phase == velerov2alpha1api.DataUploadPhaseAccepted {
return true
}
+ if du.Status.Phase == velerov2alpha1api.DataUploadPhasePrepared {
+ return true
+ }How to reproduce it (as minimally and precisely as possible)?
- Trigger a backup that uses the built-in data mover, generating a
DataUploadCR. - Simulate a dropped Kubernetes watch event right after the
DataUploadenters thePreparedphase. - Notice that the
DataUploadtask never progresses to InProgress and never times out—it remains stuck in Prepared indefinitely because the periodic reconciliation loop ignores it.
Source: velero-io/velero