Transient forbidden on TaskRun pod create (GC admission RESTMapping) is treated as a permanent failure

Expected Behavior

If creating a TaskRun pod fails with a transient API error — for example kube-apiserver GC admission returning:

pods "…-pod" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion tekton.dev/v1 Kind TaskRun: no matches for kind "TaskRun" in version "tekton.dev/v1"

the TaskRun should stay pending and the controller should retry pod creation (as it already does for ResourceQuota conflicts). Once discovery/RESTMapper recovers, the pod should be created and the TaskRun should proceed.

Actual Behavior

handlePodCreationError only requeues for quota errors. This forbidden is not quota and not SCC/PodSecurity, so it falls into the default branch: it is wrapped as controller.NewPermanentError, the TaskRun is MarkResourceFailed with PodCreationFailed, and the message is suffixed with Maybe missing or invalid Task … even when the Task is valid.

Quota conflicts requeue; this path does not:

func (c *Reconciler) handlePodCreationError(tr *v1.TaskRun, err error) error {
 switch {
 case isResourceQuotaConflictError(err):
  return controller.NewRequeueAfter(time.Second)
 case isExceededResourceQuotaError(err):
  return controller.NewRequeueAfter(time.Minute)
 case isPodAdmissionFailed(err): // only PodSecurity / SCC
  tr.Status.MarkResourceFailed(podconvert.ReasonPodAdmissionFailed, err)
 default:
  msg := fmt.Sprintf("failed to create task run pod %q: %v. Maybe missing or invalid Task ...", tr.Name, err)
  err = controller.NewPermanentError(errors.New(msg))
  tr.Status.MarkResourceFailed(podconvert.ReasonPodCreationFailed, err)
 }
 return err
}

(isPodAdmissionFailed only matches violates PodSecurity or security context constraint.)

This is still the behavior on main.

Steps to Reproduce the Problem

1. Serve tekton.dev via an aggregated APIService (CRDs not in the local kube-apiserver RESTMapper used by GC admission).
2. Briefly disrupt aggregated discovery (APIService Unavailable, or a short control-plane/konnectivity blip so the aggregator RESTMapper drops TaskRun).
3. While discovery is empty, let a TaskRun try to create its pod (ownerReferences with blockOwnerDeletion: true).
4. Observe pod create rejected with the RESTMapping forbidden above.
5. After discovery is healthy again (seconds later), the TaskRun stays Succeeded=False / PodCreationFailed and the controller logs taskrun done — it does not create the pod.

Observed on a live cluster (Tekton v1.15.0)

Controller image/version: OpenShift Pipelines, app.kubernetes.io/version=v1.15.0 (commit 7289f7d in logs).

During a ~20s hosted-apiserver/konnectivity reset, GC admission could not map tekton.dev/v1 TaskRun. Example TaskRun load-test-2-mqzmh-source-to-sdist:

Twelve TaskRuns in that window each got 1–2 create attempts, then taskrun done by 13:20:03–13:20:17. No further pod creates. Server-side dry-run of a pod with the same TaskRun ownerRef succeeded after discovery recovered.

Additional Info

• Kubernetes version: OpenShift 4.21.27 / Kubernetes v1.34.9
• Tekton Pipeline version: v1.15.0

The RESTMapping forbidden is produced by kube-apiserver GC admission when the owner kind is missing from its RESTMapper (common with aggregated APIs after a discovery gap). That condition can last seconds. Treating it like a bad Task spec makes a brief API-server issue a terminal TaskRun failure.

Suggested direction: retry forbidden errors that mention blockOwnerDeletion / RESTMapping (or, more generally, retry IsForbidden unless it is quota, PodSecurity, or SCC), instead of NewPermanentError.

Filed by Gal's Cursor.