`Consider a safer default (or explicit HA/high-concurrency guidance) for default-imagepullbackoff-timeout`

Summary

TaskRunImagePullFailed (added to fail fast on ImagePullBackOff, config default-imagepullbackoff-timeout, default 0) does not distinguish transient, self-resolving image-pull errors from permanent ones. Under bursty, high-concurrency pod creation — e.g. an HA controller deployment with a high threads-per-controller value — many pods can request an image pull from the same node within a very short window, exceeding the container runtime's image-pull rate limit (CRI-O's default imagePullQPS ~5/sec/node). The resulting ErrImagePull: pull QPS exceeded is transient and normally resolves within a few seconds via Kubernetes' own backoff/retry, but with the default default-imagepullbackoff-timeout: 0, the controller kills the TaskRun on the first occurrence, before that retry can happen.

Reproduction

• OpenShift Pipelines 1.23.0 / Tekton Pipelines v1.12.0 (commit 7ce6b655b0a061a8bdca75d3d5e18b7140f90ea3), but the code path (taskrun/taskrun.go PullImageFailed detection → TaskRunImagePullFailed stop) is unchanged upstream.
• Controller config: 10 replicas (HA, Deployments + lease-based leader election), threads-per-controller: 32, kube-api-qps: 50, kube-api-burst: 50.
• Workload: 1000 PipelineRuns, 4 parallel TaskRuns each, all using registry.access.redhat.com/ubi8-micro:latest (so imagePullPolicy defaults to Always), 20 concurrent PipelineRuns across 5 namespaces.
• Cluster: 5 worker nodes, CRI-O default image-pull QPS.

Observed

• 34.5% PipelineRun failure rate at concurrency 20 (345/1000).

• Controller logs: 2,093 ErrImagePull: pull QPS exceeded occurrences and 460 TaskRunImagePullFailed stops in a single 10-minute run.

• Cross-configuration isolation test (identical scenario/image/cluster, only HA-replica-count and thread-count varied):

  Configuration	Total reconciler threads	Workqueue depth (mean)	TaskRun→Pod lag (mean)	PipelineRun failures (conc 20)	pull QPS exceeded count
  Standard (1 replica, 2 threads)	2	525	69.7s	0	0
  HA only (10 replicas, 2 threads)	20	249	2.0s	0	0
  QBT only (1 replica, 32 threads)	32	196	6.6s	0	0
  HA + QBT (10 replicas, 32 threads)	320	9.5	0.18s	334	2,093

  Only the HA+QBT configuration drains the workqueue fast enough (320 concurrent reconcilers) to create pods in near-instantaneous bursts (~80 pods within ~0.5s) instead of being naturally paced by workqueue backlog. That burst is what exceeds CRI-O's per-node pull QPS. Controller CPU/memory, API server, etcd, and worker node resources are all well within limits in every configuration — this is not a resource-exhaustion issue.

• Failed-PipelineRun duration is consistently ~18-19s, matching the timing of a single Kubernetes image-pull backoff cycle — i.e., these TaskRuns would very likely have succeeded on Kubernetes' own next retry had they not been terminated first.

• Same behavior reproduced on Tekton v1.22 (192 failures at concurrency 20, before TaskRunImagePullFailed existed) purely from CRI-O QPS exhaustion without controller-side retries being cut short — confirming the burst/QPS mechanism is independent of this feature, but the feature makes it substantially worse (79.7% more failures on v1.23 at the same concurrency).

Expected

A single transient pull QPS exceeded (or other clearly-recoverable) image-pull error should not immediately terminate a TaskRun via TaskRunImagePullFailed. Either:

1. Classify known-transient error substrings (pull QPS exceeded, 502, 504, i/o timeout, TLS handshake timeout, connection refused) separately from permanent ones (invalid image name, auth denied) and only fail-fast on the latter, or
2. Reconsider the default value of default-imagepullbackoff-timeout (currently 0) so that some retry window is given by default rather than opt-in via config.

Related issues

• #5987 (closed) — added the default-imagepullbackoff-timeout config this feature relies on; that config exists but defaults to 0 (immediate fail) and doesn't distinguish error types.
• #8357 (open) — reports default-imagepullbackoff-timeout being inconsistently honored depending on whether entrypoint inference is needed; different code path, same symptom family (configured grace period not protecting against a transient failure).
• #6530 (open) — different rate limit (Kubernetes API QPS during tag→digest resolution, not CRI-O image-pull QPS) but the same underlying pattern: high-concurrency reconciliation exhausting a rate limiter with no graceful backoff.

Additional context

Full root-cause analysis and cross-configuration data available on request — happy to share the raw controller logs, workqueue depth metrics, and per-configuration failure counts referenced above.