boundServiceAccountToken refresh permanently fails when the scaler cache captures a cancelled request context

Report

KEDA can permanently stop refreshing a boundServiceAccountToken, leaving the affected ScaledObject unable to read metrics until keda-operator is restarted.

The scaler cache stores a Factory closure that captures whatever context.Context was live when the cache was built. If the cache happens to be built while serving an HPA metrics read, that context is the gRPC request context, which the apiserver cancels as soon as the response is written. About an hour later, when the token expires, the refresh runs under that cancelled context, CreateToken fails, and an empty token is returned — surfacing as a misleading metadata-validation error. It never recovers, because the refresh re-stores the same closure.

This is a context-lifetime bug in pkg/scaling, not specific to any one scaler. It affects any trigger authenticated with boundServiceAccountToken, since a full scaler rebuild is the only way to mint a fresh token. We encountered it with the Datadog scaler in Cluster Agent proxy mode: six ScaledObjects sat pinned to their fallback replica counts for ~13h.

Expected Behavior

KEDA refreshes a boundServiceAccountToken when it expires, and the ScaledObject continues scaling.

Actual Behavior

KEDA can permanently fail to refresh the token because the rebuild runs under a context.Context that was cancelled up to an hour earlier. The ScaledObject then reports error parsing Datadog metadata: BearerToken is required on every poll and never recovers. Only restarting keda-operator clears it. In our cluster six ScaledObjects sat pinned to their fallback replica counts for ~13h.

The root cause is a context lifetime bug, not a Datadog-specific one: any scaler using boundServiceAccountToken is affected, because a full Scaler rebuild is the only way to mint a fresh token.

Steps to Reproduce the Problem

Mechanism. buildScalers creates a Factory closure that captures its enclosing ctx (scalers_builder.go#L49) and stores it in ScalerBuilder. refreshScaler later invokes oldSb.Factory() with no arguments (scalers_cache.go#L264); its own live ctx is used only to Close() the old scaler. It then re-stores the same closure (#L272), so the captured context survives every subsequent refresh.

This is benign while the captured context is the reconcile/scale-loop one, which outlives the cache. It is fatal when the cache is built while serving an HPA read, because that context is the gRPC request context and the apiserver cancels it as soon as the response is written.

How the cache comes to be built on the read path. getScalersCacheForScaledObject passes a nil generation (scale_handler.go#L428), so the read path only builds when the entry is absent. Any scaler error that survives refreshScaler's retry evicts the whole entry (#L757); the next caller rebuilds, and an HPA read frequently wins that race since it polls more often than KEDA's default 30s pollingInterval.

Sequence:

1. A transient backend error survives the retry → ClearScalersCache evicts the entry.
2. An HPA read hits the miss → buildScalers captures the gRPC request context into every trigger's Factory. The rebuild succeeds and serves metrics normally.
3. The apiserver cancels that context when the response is written.
4. ~1h later the token expires → refreshScaler → Factory() under the dead context → CreateToken fails → empty token returned (scale_resolvers.go#L751) → BearerToken is required.
5. Permanent: the same closure is re-stored on every refresh.

Note the ~1h gap between the trigger and the symptom, and that step 2 succeeds — which is why this is hard to attribute in the field.

Deterministic repro without the race: a unit test can construct a ScalerBuilder whose Factory captured a context, cancel it, call refreshScaler with a live context, and observe the rebuild use the cancelled one. Happy to contribute this alongside a fix.

Our configuration: Datadog scaler in Cluster Agent proxy mode (useClusterAgentProxy: "true"), authenticated with a ClusterTriggerAuthentication using boundServiceAccountToken and authMode: bearer. 19 ScaledObjects with Datadog triggers, 22 Datadog triggers total, all on the default 30s polling interval.

Logs from KEDA operator

The two lines appear together. The first is the real failure; the second is what it gets reported as, which is misleading since nothing is wrong with the user's metadata:

error trying to create bound service account token for service account
  client rate limiter Wait returned an error: context canceled
error parsing Datadog metadata: BearerToken is required

KEDA Version

2.20.1

Kubernetes Version

1.35

Platform

Amazon Web Services

Scaler Details

Datadog (Cluster Agent proxy mode)

Would you be open to contributing a fix?

Yes

Anything else?

Running on EKS, Kubernetes 1.35.6.

Searched existing issues before filing; the closest match is #5428, which covers the same code path but was closed without resolution.

Confirmed still present on main.

Proposed fix — pass the caller's context into the Factory. One type, one call site, one production constructor:

- Factory func() (scalers.Scaler, *scalersconfig.ScalerConfig, error)
+ Factory func(ctx context.Context) (scalers.Scaler, *scalersconfig.ScalerConfig, error)

// scalers_builder.go:49
- factory := func() (scalers.Scaler, *scalersconfig.ScalerConfig, error) {
+ factory := func(ctx context.Context) (scalers.Scaler, *scalersconfig.ScalerConfig, error) {

// scalers_cache.go:264
- newScaler, sConfig, err := oldSb.Factory()
+ newScaler, sConfig, err := oldSb.Factory(ctx)

Possibly related: #5428 (same path; a maintainer noted "cancelled by the caller", closed unresolved), #5926, #6738, #6359.

I'm happy to submit a PR with the fix and a regression test.