ResourceTracker GC deletes managed resources after rapid A -> B -> A spec updates

Describe the bug

After an Application spec was changed from A to B and quickly reverted to A, KubeVela deleted the Application's only ResourceTracker (RT) and all resources tracked by it, including its Deployment, Service, ServiceAccount, and HPA.

The Application itself was not deleted. It continued to report running and healthy: true, with the workflow marked as finished, but none of its managed resources remained. The resources were not recreated until a later spec change triggered a new workflow run.

Our analysis suggests that this happens because workflow restart detection and ResourceTracker history detection use different identifiers:

1. Workflow re-execution is keyed by ApplicationRevision name + Application spec hash.
2. A ResourceTracker is considered current only when rt.spec.applicationGeneration == application.metadata.generation.

These identifiers can diverge when two spec updates are processed by the API server before the controller reconciles the intermediate state:

• The A -> B -> A updates advance metadata.generation twice.
• The final spec is identical to the previously reconciled spec, so the workflow identity is unchanged and the workflow is not restarted.
• Because component dispatch is skipped, no RT is created for the new generation.
• The existing RT is classified as history because its applicationGeneration no longer matches the Application generation.
• Normal RT garbage collection deletes that RT. Its finalizer then deletes the resources it tracks.

There appears to be no guard on this GC path for a non-deleting Application that has history RTs but no current RT.

To Reproduce

The incident was triggered by an external GitOps controller that changed the Application spec from A to B and back to A within a few seconds. The Application used standard built-in component/trait definitions; the specific definitions do not appear to matter, only the sequence of spec updates.

1. Apply an Application with spec A at generation N.
2. Wait for its workflow to finish and confirm that an RT for generation N exists and tracks the managed resources.
3. Change the Application spec to B.
4. Before KubeVela reconciles the intermediate spec, change it back to A.
5. Wait for KubeVela to reconcile the Application.

The timing requirement in step 4 is important: both spec updates must reach the API server before KubeVela reconciles spec B. We have not verified this ourselves, but it should be possible to remove the timing dependency by scaling the vela-core deployment to zero replicas, applying both spec updates, and then restoring the deployment to its original replica count.

What happens after step 5:

• application.metadata.generation is N+2.
• The workflow identity is the same as at generation N, because the final spec is again A.
• The existing workflow remains Finished, so component dispatch is skipped.
• No RT is created for generation N+2.
• The generation-N RT is classified as history and garbage-collected.
• The RT finalizer deletes all resources tracked by it.
• The Application continues to report running, healthy: true, and a finished workflow, but does not recreate the resources.

Expected behavior

For an Application that is not being deleted, KubeVela should not delete history RTs when no current-generation RT exists.

It should preserve the existing RT and either:

• return an error so that reconciliation is retried, while keeping the existing resources intact; or
• restart the workflow and create a current-generation RT before deleting any history RTs.

Screenshots

Not applicable. Kubernetes audit log evidence from the incident is included under Additional context below.

KubeVela Version

• Observed on v1.5.2 (kubevela/git-360f69b).
• We have not reproduced the issue on master. However, code inspection at a59bd39e725ba35952cf828458143e4b2b0c4dd6 suggests that the same condition may still be possible: workflow restart detection uses the revision and spec hash, current RT detection uses generation, and the normal GC path can delete history RTs without checking that a current RT exists.

Cluster information

• Kubernetes: v1.33 (Amazon EKS)
• The Application spec updates were made by an external GitOps controller running in the same cluster.

Additional context

Relevant code paths

The following references are from v1.5.2. Master has a similar structure at the commit noted above.

Workflow restart detection

pkg/workflow/workflow.go builds the workflow identity from the ApplicationRevision name and Application spec hash. A finished workflow is not restarted when that identity is unchanged:

specHash, err := utils.ComputeSpecHash(app.Spec)
version = fmt.Sprintf("%s:%s", rev, specHash)

if w.app.Status.Workflow == nil ||
   w.app.Status.Workflow.AppRevision != revAndSpecHash {
    return w.restartWorkflow(ctx, revAndSpecHash)
}
if wfStatus.Finished {
    return common.WorkflowStateFinished, nil
}

Because the final spec is again A, revAndSpecHash is unchanged and the workflow is not restarted.

Lazy creation of the current RT

pkg/resourcekeeper/resourcekeeper.go creates the current RT lazily:

func (h *resourceKeeper) getCurrentRT(ctx context.Context) (...) {
    if h._currentRT == nil {
        h._currentRT, err =
            resourcetracker.CreateCurrentResourceTracker(...)
    }
    return h._currentRT, nil
}

When the workflow remains Finished, apply-component and component dispatch are skipped. As a result, getCurrentRT() is not called and no RT is created for the new Application generation.

Current and history RT detection

pkg/resourcetracker/app.go derives the current RT name from the Application generation and classifies RTs by comparing applicationGeneration with the current generation:

func getCurrentResourceTrackerName(app *Application) string {
    return fmt.Sprintf("%s-v%d-%s", app.Name, app.GetGeneration(), app.Namespace)
}

if rt.Spec.ApplicationGeneration == app.GetGeneration() {
    currentRT = rt
} else {
    historyRTs = append(historyRTs, rt)
}

When the existing RT has applicationGeneration = N and the Application has generation = N+2, the RT is placed in historyRTs and currentRT is nil.

GC after a finished workflow

pkg/controller/core.oam.dev/v1alpha2/application/application_controller.go continues to ResourceTracker GC after WorkflowStateFinished unless the workflow was terminated:

case common.WorkflowStateFinished:
    logCtx.Info("Workflow state=Finished")
    if status := app.Status.Workflow; status != nil && status.Terminated {
        return r.result(nil).ret()
    }
    // ...continues to garbage collection...
    return r.gcResourceTrackers(logCtx, handler, phase, true, true)

In the normal GC path in pkg/resourcekeeper/gc.go, history RTs are treated as inactive and deleted:

} else {
    inactiveRTs = h._historyRTs
}
// ...
for _, rt := range inactiveRTs {
    if rt != nil && rt.GetDeletionTimestamp() == nil {
        h.Client.Delete(ctx, rt)
    }
}

We could not find a check on this path that prevents deletion when currentRT == nil. A similar check exists in GarbageCollectLegacyResourceTrackers, but that is the legacy RT migration path rather than the normal versioned RT GC path used here.

Proposed fix

One possible fail-safe is to stop normal RT garbage collection when all of the following are true:

• the Application is not being deleted;
• no current RT exists; and
• at least one history RT exists.

For example (pseudo-code):

if app.GetDeletionTimestamp() == nil &&
   currentRT == nil &&
   len(historyRTs) > 0 {
    return errorOrRequeue(
        "current ResourceTracker is missing; refusing to GC history",
    )
}

This would preserve the managed resources while surfacing the inconsistent state. Another option would be to restart the workflow and create a current RT before allowing history RTs to be collected.

Audit log evidence (anonymized)

Environment-specific names are replaced with <app> and <ns>. Generation numbers and event ordering are preserved. Times are UTC.

ApplicationRevision names use the revision number, while ResourceTracker names use metadata.generation, so the v numbers below differ between the two resource types.

05:27:30  KubeVela  create ApplicationRevision  <app>-v370
05:27:32  KubeVela  create ResourceTracker      <app>-v349-<ns>   (applicationGeneration = 349)
05:27:34  KubeVela  delete old RT               <app>-v348-<ns>

# External GitOps controller changes the spec from A to B and back to A:
05:32:28  GitOps    patch Application  <app>          (spec -> B; generation 349 -> 350)
05:32:37  GitOps    patch Application  <app>          (spec -> A; generation 350 -> 351)

05:32:43  KubeVela  delete ResourceTracker  <app>-v349-<ns>   (the only RT)
05:33:00  KubeVela  delete Deployment       <app>
05:33:00  KubeVela  delete Service          <app>
05:33:00  KubeVela  delete ServiceAccount   <app>
05:33:00  KubeVela  delete VirtualService   <app>
                     (also: HPA, DestinationRule, PDB, EnvoyFilter,
                      TargetGroupBinding, ExternalSecret)
05:33:17  KubeVela  patch Application status (RT finalizer processing complete)

The actor and user agent for each delete event were:

user:      system:serviceaccount:vela-system:vela-core
userAgent: kubevela/git-360f69b

There is no create event for an RT at generation 351 in the audit log. This is consistent with no current RT being created before the generation-349 RT was deleted along with its tracked resources.

The Application survived and continued to report running and healthy: true. It recovered only after a later spec change triggered a new workflow run and created a new current RT.