Providers sharing a DeploymentRuntimeConfig contend over the ServiceAccount's ownerReferences, causing an unbounded reconcile loop

What happened?

If two or more Providers reference the same DeploymentRuntimeConfig, and that config pins serviceAccountTemplate.metadata.name, their ProviderRevisions end up in a permanent tug-of-war over the ServiceAccount's ownerReferences. Each revision's package-runtime reconcile overwrites the owner with itself, that write wakes the others, and they all reconcile and re-claim it. It never converges.

It's completely silent. No events, no log lines, no errors — every reconcile returns result="success", so the workqueue rate limiter never backs off either. We only found it because it showed up on a cloud bill: the API server on this cluster is reached over a public endpoint, so the write storm was being billed as load balancer data processing. That's what sent us looking.

Two clusters, same config:

controller_runtime_reconcile_total{controller="package-runtime/providerrevision.pkg.crossplane.io",result="success"}
  cluster A:    718,850  over 5.27h    (~38/sec)
  cluster B: 13,624,307  over 131.7h   (~29/sec)

Each pass server-side-applies the revision's runtime objects, which works out to ~190 PATCH/sec against the API server:

rest_client_requests_total{code="200",method="PATCH"}  3,614,212
rest_client_requests_total{code="200",method="GET"}    3,604,791

Broken down by resource, it's exactly the objects package-runtime manages, in a 2:1:1:1 ratio (two TLS secrets, then Service, Deployment, ServiceAccount):

secrets           38.9%
services          20.2%
deployments.apps  20.0%
serviceaccounts   19.4%

None of it changes anything — every one of those objects has a stable resourceVersion. It's ~250 GB/day of no-op writes.

You can watch the ownership move just by reading the ServiceAccount in a loop:

$ for i in $(seq 1 10); do kubectl -n crossplane-system get sa provider-gcp \
    -o jsonpath='{.metadata.ownerReferences[0].name}{"\n"}'; done
upbound-provider-family-gcp-2e33bfd0f501
upbound-provider-family-gcp-2e33bfd0f501
upbound-provider-family-gcp-2e33bfd0f501
upbound-provider-gcp-cloudplatform-f1fe8bc55c47
upbound-provider-family-gcp-2e33bfd0f501
upbound-provider-gcp-cloudplatform-f1fe8bc55c47
upbound-provider-gcp-cloudplatform-f1fe8bc55c47
upbound-provider-family-gcp-2e33bfd0f501
upbound-provider-family-gcp-2e33bfd0f501
upbound-provider-gcp-cloudplatform-f1fe8bc55c47

There's never more than one entry in the list. A ServiceAccount belonging to a single revision (e.g. any function's) is rock stable by comparison, and a DeploymentRuntimeConfig referenced by exactly one Provider is fine too — we have one of those in the same cluster and it never moves.

The relevant code looks to be ServiceAccountWithOwnerReferences in internal/controller/pkg/runtime/runtime_override_options.go:

func ServiceAccountWithOwnerReferences(owners []metav1.OwnerReference) ServiceAccountOverride {
	return func(sa *corev1.ServiceAccount) {
		sa.OwnerReferences = owners
	}
}

which runtime.go calls with a single-element slice built from meta.AsController(...).

What did you expect to happen?

That several Providers can share one DeploymentRuntimeConfig and the ServiceAccount it names, and that this settles into a steady state.

Sharing is the whole reason the field is useful to us, because the default name isn't usable for cloud identity federation. Crossplane names the service account after the revision, and that name carries a content hash:

crossplane-contrib-function-auto-ready-a37a591901b6

which changes on every version bump. Anything bound to system:serviceaccount:crossplane-system:<name> — an Azure federated credential, a GCP workload identity binding — breaks on each provider upgrade. Pinning a stable name is the natural way around that, and pointing a whole provider family at one config is the natural way to do it once rather than N times. Upbound ships Azure, AWS and GCP as families of a dozen-odd providers, so this isn't an unusual shape.

How can we reproduce it?

1. Create a DeploymentRuntimeConfig with a fixed service account name:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-gcp
spec:
  serviceAccountTemplate:
    metadata:
      name: provider-gcp

1. Install two providers from the same family, both pointing at it:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: upbound-provider-family-gcp
spec:
  package: xpkg.upbound.io/upbound/provider-family-gcp:v2.6.0
  runtimeConfigRef:
    name: provider-gcp
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: upbound-provider-gcp-cloudplatform
spec:
  package: xpkg.upbound.io/upbound/provider-gcp-cloudplatform:v2.6.0
  runtimeConfigRef:
    name: provider-gcp

1. Watch sa/provider-gcp's ownerReferences[0].name move between the two revisions, and controller_runtime_reconcile_total for package-runtime/providerrevision.pkg.crossplane.io climb without limit.

Two providers is enough to trigger it; it gets proportionally worse with more.

What environment did it happen in?

Crossplane version: v2.3.3 Kubernetes version: v1.35.6 (AKS) Cloud provider: Azure