Resolver-based PipelineRef still causes unbounded pipeline label cardinality (gap left by #9826)
Expected Behavior
For a PipelineRun that references its Pipeline via a resolver (git, bundle, hub, etc.), the pipeline label on tekton_pipelines_controller_pipelinerun_* / tekton_pipelines_controller_taskrun_* metrics should reflect a stable identifier for the pipeline (e.g. derived from generateName, matching the fix applied in #9826 for inline PipelineSpec), so that all runs of the same pipeline aggregate under the same metric series.
Actual Behavior
For any pipelineRef.resolver other than cluster (i.e. git, bundle, hub), propagatePipelineNameLabelToPipelineRun in pkg/reconciler/pipelinerun/pipelinerun.go falls through to:
case pr.Spec.PipelineRef != nil && pr.Spec.PipelineRef.Resolver != "":
pr.ObjectMeta.Labels[pipeline.PipelineLabelKey] = pr.NameThis sets the pipeline label to the PipelineRun's own unique instance name instead of a stable pipeline identifier. Only the cluster resolver's kind=pipeline/name=X param convention (added in #7051, 2023, unrelated to #9826) gets a stable-name override.
At metrics.pipelinerun.level=pipeline, this is the only identifying label attached to the metric (pipelineInsertTag in pkg/pipelinerunmetrics/metrics.go explicitly discards the pipelinerun argument), so every execution of the same pipeline creates a brand-new, permanent time series instead of aggregating into one. Combined with there being no cleanup mechanism for old series (neither pipelinerunmetrics nor taskrunmetrics registers a delete/unregister handler - only a periodic running-count gauge loop), this causes unbounded cardinality growth over time, eventually leading to scrape failures against the controller's /metrics endpoint.
Observed live: 10,000+ unique pipeline label values on tekton_pipelines_controller_pipelinerun_duration_seconds_count, including 24+ distinct values for a single pipeline (yarn-e2e-test) alone - one per run, e.g. yarn-e2e-test-hqrwn, yarn-e2e-test-jxd7w, yarn-e2e-test-7tqmr, etc.
Steps to Reproduce the Problem
- Set metrics.pipelinerun.level: "pipeline" in the config-observability ConfigMap.
- Create and run a PipelineRun that references its Pipeline via a non-cluster resolver, e.g.:
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
generateName: example-pipeline-
spec:
pipelineRef:
resolver: git
params:
- name: url
value: https://github.com/<org>/<repo>.git
- name: revision
value: main
- name: pathInRepo
value: pipeline.yaml- Run it multiple times (or trigger repeatedly, e.g. via an Integration Test Scenario / Pipelines-as-Code integration test), then query:
curl http://<controller>:9090/metrics | grep tekton_pipelines_controller_pipelinerun_duration_seconds_countObserve a distinct pipeline="" label per run instead of a single shared value.
Additional Info
- Kubernetes version:
Client Version: 4.22.8
Kustomize Version: v5.7.1
Server Version: 4.19.42
Kubernetes Version: v1.32.13- Tekton Pipeline version:
Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'
v1.15.0Related:
- #9140 — original enhancement request this pattern was meant to address
- #9826 — fixed the PipelineSpec branch, left the PipelineRef.Resolver branch unfixed
- #7051 (2023) — added the cluster-resolver-specific override, predates and is unrelated to #9826
Source: tektoncd/pipeline