Baike.dev
Log in
Back to tool/Back to issues
#10610·pipeline

Support Kubernetes v1.36 Server-Side Sharded List and Watch in Pipelines Controllers

Author: gbenhaimCreated Aug 16, 2026Updated Sep 5, 2026
Labelskind/featurehelp wanted

Feature Request

Summary

Kubernetes v1.36 introduced Server-Side Sharded List and Watch as an alpha feature (KEP-5866). This feature allows the API server to filter watch events at the source, so that each controller replica only receives objects within its assigned hash range — rather than the full event stream.

Tekton Pipelines controllers (e.g., the PipelineRun and TaskRun reconcilers) would benefit significantly from adopting this capability, especially in large-scale environments where thousands of PipelineRuns and TaskRuns are created concurrently.

Problem

Today, when the Tekton Pipelines controller is scaled horizontally (multiple replicas), every replica receives the full stream of watch events for all PipelineRuns, TaskRuns, and other resources. Each replica then deserializes and processes every event, only to discard objects it isn't responsible for (via leader election or other partitioning logic). This has several consequences:

  • N replicas × full event stream: CPU and memory cost scales with replica count, not shard size.
  • Network bandwidth scales with replicas, not with the actual work each replica performs.
  • Wasted deserialization cost: each replica spends CPU deserializing events it will immediately discard.

In clusters running thousands of concurrent PipelineRuns (common in CI/CD platforms like Konflux), this becomes a significant bottleneck for etcd, the API server, and the controllers themselves.

Proposed Solution

Add support for the ShardSelector field in ListOptions when configuring informers in the Tekton Pipelines controller. When running on Kubernetes >= 1.36 with the ShardedListAndWatch feature gate enabled, each controller replica would:

  1. Be assigned a portion of the hash space (e.g., via an environment variable, ordinal index, or a flag).
  2. Inject a shardSelector into its informer ListOptions using WithTweakListOptions, so the API server only sends events for objects within that replica's hash range.

Example for a 2-replica deployment:

go
// Replica 0: lower half of the hash space
"shardRange(object.metadata.uid, '0x0000000000000000', '0x8000000000000000')"

// Replica 1: upper half of the hash space
"shardRange(object.metadata.uid, '0x8000000000000000', '0x10000000000000000')"

The controller should fall back gracefully to the current behavior (full watch stream) when:

  • The API server doesn't support the feature (i.e., shardInfo is absent from the list response).
  • The feature gate is not enabled.
  • The controller is running as a single replica.

Expected Benefits

  • Reduced API server and etcd load: each replica only receives its shard of watch events, reducing the total data the API server must serialize and send.
  • Linear scaling: adding controller replicas now divides the work rather than multiplying the watch cost.
  • Lower memory usage: each replica's informer cache only holds objects in its shard.
  • Better performance at scale: critical for environments running thousands of concurrent PipelineRuns.

Multiplier Effect with API Server Sharding (kube-shard)

This feature becomes a force multiplier when combined with kube-shard — an aggregated secondary API server backed by PostgreSQL (via Kine) instead of etcd. kube-shard offloads Tekton CRDs (PipelineRuns, TaskRuns, etc.) from the primary cluster's etcd to a dedicated API server with no storage size constraint, eliminating the 8 GB etcd limit that caps PipelineRun concurrency.

The key insight is that with kube-shard, the number of secondary API server replicas can scale proportionally to the number of Tekton Pipelines controller replicas. In a standard cluster, scaling the API server is constrained by the shared etcd cluster, but with kube-shard the secondary API server is backed by PostgreSQL, which scales independently. This means:

  • Without sharded watches: scaling the Pipelines controller to N replicas puts N× the watch load on the API server (every replica watches everything). Scaling the API server helps with serving capacity but does not reduce the per-replica event volume.
  • With sharded watches + kube-shard: each of the N controller replicas only receives 1/N of the events, and the secondary API server can be scaled to N replicas to match — each API server replica serving a proportional share of the sharded watch streams. The controller-side and API-server-side scaling reinforce each other rather than working against each other.

In practice, for a deployment with 10 Pipelines controller replicas processing thousands of concurrent PipelineRuns:

  • Each controller replica watches only 1/10 of the objects (via shardSelector).
  • The secondary API server can run 10 replicas, each handling a fraction of the watch connections.
  • PostgreSQL handles the aggregate read/write load independently of the controller replica count.

This turns horizontal scaling from an additive cost (more replicas = more total watch traffic) into a multiplicative benefit (more replicas = more throughput with constant per-replica cost).

Additional Context

This would be particularly impactful for Tekton deployments in multi-tenant CI/CD platforms where PipelineRun volumes are high and controller scalability is a key concern.

This issue was opened by Gal's Cursor.

Source: tektoncd/pipeline

View original on GitHubView discussion on GitHub