Workqueue depth grows indefinitely after XRD update due to stale controller cache

Observed on Crossplane versions v2.2.1 and v2.3.1.

What happened?

After an XRD generation change, composite resource controllers repeatedly reconcile resources using stale cached data. The resource versions observed during reconciliation never change, indicating the controller runtime cache is not receiving UPDATE events from the Kubernetes API server.

Functionally the compositions continue working but the workqueue_depth metric stays elevated and "not yet ready" events continue to be placed on XRs.

Expected behavior:

• After pod restart, informer caches are populated with fresh data from etcd
• Resource versions update as resources are modified
• Workqueue depth decreases as resources are successfully reconciled
• Composed resources that become ready are detected and reflected in parent resource status

Actual behavior:

• Controllers reconcile resources using the same stale resource versions repeatedly
• Workqueue depth grows and never decreases
• Composed resources perpetually show "not yet ready" even if they may have become ready
• No errors visible in INFO logs - only detectable with DEBUG logging enabled
• Metrics show elevated workqueue_depth that never clears and a matching increase in controller_runtime_reconcile_errors_total

How can we reproduce it?

Prerequisites:

• Crossplane v2.3.1
• Any composite resources created (XR with composed resources)
• Debug logging enabled for detailed observation
• Metrics collection enabled to observe workqueue_depth (e.g. port-forward on metrics port + curl + grep)

Reproduction Steps:

1. Create composite resources:

   bash复制

   # Create any XRD and composition with composed resources
   kubectl apply -f your-xrd.yaml
   kubectl apply -f your-composition.yaml
   kubectl apply -f your-xr-instance.yaml

2. Verify resources are reconciling:

   Check workqueue depth is 0 or low

3. Apply a change to the XRD:

   This can be a change to the spec for example.

4. Observe the issue:

   Enable debug logging and watch for repeating resource versions:

   bash复制

   kubectl logs -n crossplane-system -l app=crossplane -f --tail=100 | grep "Successfully composed"

   Monitor workqueue depth: this should now read a number equal to the number of XRs.

5. Expected vs Actual:

   • Expected: Resource versions change with each reconciliation, workqueue depth returns to 0
   • Actual: Same resource versions appear repeatedly, workqueue depth stays elevated (equals number of affected resources)

Additional Evidence:

Debug logs show index conflict error (showing cache wasn't cleared):

cannot add composite GVK index: indexer conflict: map[field:compositeResourcesRefs:{}]

This error is silently logged at DEBUG level in the code at: internal/controller/apiextensions/definition/reconciler.go:576

Root Cause Analysis

Code Investigation:

The bug is in the controller engine's cache lifecycle management:

1. Controller Stop Method (internal/engine/engine.go:372-402):

func (e *ControllerEngine) Stop(ctx context.Context, name string) error {
    // Stops watches ✅
    for wid, w := range c.sources {
        if err := w.Stop(ctx); err != nil {
            return errors.Wrapf(err, "cannot stop %q watch for %q", wid.Type, wid.GVK)
        }
        delete(c.sources, wid)
    }
    
    // Cancels context ✅
    c.cancel()
    delete(e.controllers, name)
    
    // Does NOT clear cache ❌
    // Does NOT remove field indexes ❌
    // Does NOT force informer resync ❌
    
    return nil
}

2. Controller Restart on XRD Change (internal/controller/apiextensions/definition/reconciler.go:500-514):

if ControllerNeedsRestart(d) {
    r.record.Event(d, event.Normal(reasonRestartXR, "XRD specification changed; restarting controller to apply updates"))
    
    // Stops controller but doesn't invalidate cache
    if err := r.engine.Stop(ctx, composite.ControllerName(d.GetName())); err != nil {
        return reconcile.Result{}, err
    }
    
    // Immediately starts new controller with SAME stale cache
    // No cache invalidation between Stop() and Start()
}

3. Index Conflict Silent Error (internal/controller/apiextensions/definition/reconciler.go:574-576):

if err := r.engine.GetFieldIndexer().IndexField(ctx, u, compositeResourcesRefsIndex, IndexCompositeResourcesRefs(schema)); err != nil {
    r.log.Debug(errAddIndex, "error", err)  // ❌ Only DEBUG, should be ERROR
}

Index already exists from previous controller instance, proving cache state wasn't cleaned up.

Why The Cache Becomes Stale:

1. Pod restarts → Controllers start fresh
2. Informers perform LIST to populate cache
3. Cache populated with current resource versions from etcd
4. Watches established to receive UPDATE events
5. BUT: Informer appears to use stale list data or doesn't properly receive/process UPDATE events for previously-cached resources
6. Controllers reconcile using cached data, find composed resources "not ready"
7. Requeue for later reconciliation
8. Next reconciliation reads SAME stale cache data
9. Infinite loop - workqueue never drains

What environment did it happen in?

Crossplane version: v2.3.1 (also confirmed in v2.2)

Observed on:

• XRD generation changes
• Potentially any scenario that recreates controllers

Proposed Solution

Option 1: Force Cache Invalidation on Controller Restart

Add cache invalidation in internal/engine/engine.go after stopping a controller:

func (e *ControllerEngine) Stop(ctx context.Context, name string) error {
    // ... existing stop logic ...
    
    // NEW: Force cache invalidation for all GVKs this controller was watching
    for wid := range c.sources {
        if err := e.InvalidateCacheFor(ctx, wid.GVK); err != nil {
            e.log.Debug("cannot invalidate cache after controller stop", "error", err, "gvk", wid.GVK)
        }
    }
    
    c.cancel()
    delete(e.controllers, name)
    return nil
}

// NEW method
func (e *ControllerEngine) InvalidateCacheFor(ctx context.Context, gvk schema.GroupVersionKind) error {
    // Get informer and force resync
    // Implementation depends on informer interface
}

Option 2: Treat Index Conflicts as Errors

In internal/controller/apiextensions/definition/reconciler.go:574-576:

if err := r.engine.GetFieldIndexer().IndexField(ctx, u, compositeResourcesRefsIndex, IndexCompositeResourcesRefs(schema)); err != nil {
    // Change from debug to error and fail reconciliation
    err = errors.Wrap(err, errAddIndex)
    r.record.Event(d, event.Warning(reasonEstablishXR, err))
    return reconcile.Result{}, err  // Fail instead of silently continuing
}

This surfaces cache state issues instead of hiding them.

Option 3: Remove and Re-add Indexes on Restart

Before adding index, explicitly remove it if it exists:

// Remove existing index if present
_ = r.engine.GetFieldIndexer().RemoveIndexField(ctx, u, compositeResourcesRefsIndex)

// Then add fresh index
if err := r.engine.GetFieldIndexer().IndexField(...); err != nil {
    return reconcile.Result{}, err
}

Priority Request: This bug requires a Crossplane pod restart in every affected cluster to have the work queue processed successfully. It would be valuable to have this addressed in an upcoming patch release.