Application controller: watch resync timer has no jitter, causes thundering herd on large multi-cluster deployments

Describe the bug

ARGOCD_CLUSTER_CACHE_WATCH_RESYNC_DURATION controls how often gitops-engine's clusterCache.watchEvents relists and restarts each "group/kind" watch (pkg/cache/cluster.go, time.NewTimer(c.watchResyncTimeout)). This timer uses a fixed duration with no jitter/randomization.

On a deployment managing a large number of clusters, each watching on the order of 100 resource kinds, the application-controller pod starts all of these watches within a short startup window (e.g. right after a controller restart). Because the resync timer has no jitter, all watches keep firing their relist+rewatch in lockstep every watchResyncTimeout thereafter, causing a recurring synchronized spike of concurrent List+Decode calls instead of a steady, spread-out load.

We confirmed this via CPU profiling (pprof): the List+JSON-decode call path dominates CPU time in bursts (roughly half to most of sampled CPU time depending on when the sample lands), and go_goroutines from /metrics swings by double-digit percentages between samples taken only a few minutes apart — consistent with a synchronized burst pattern rather than steady load.

We increased ARGOCD_CLUSTER_CACHE_WATCH_RESYNC_DURATION from the default 10m to 30m hoping to reduce relist frequency and controller CPU usage. After about 48 hours of observation, average controller CPU did not improve, and CPU/goroutine samples became more erratic (larger variance between consecutive samples) — consistent with a thundering-herd pattern rather than a genuine reduction in load.

To Reproduce

1. Run argocd-application-controller against a deployment with a large number of clusters, each with on the order of 100+ resource kinds (enough that total watch count reaches tens of thousands).
2. Restart the controller (or observe right after startup).
3. Take repeated pprof CPU profiles (/debug/pprof/profile) a few minutes apart, or sample go_goroutines from /metrics a few minutes apart.
4. Observe that the List/JSON-decode CPU share and goroutine counts vary widely between samples, correlating with a periodic relist burst rather than a steady baseline, because all watches were started around the same time and their resync timers are perfectly synchronized with no jitter.

Expected behavior

Watch resync timers should include a small random jitter (similar to the common Kubernetes pattern via wait.Jitter) so that after a bulk restart, individual watches' resync moments gradually spread out over time instead of staying locked in sync indefinitely, avoiding recurring synchronized load spikes.

Version

Confirmed present in argocd v2.6.7, and also confirmed present in the current master branch (monorepo gitops-engine/pkg/cache/cluster.go, watchEvents function) as of 2026-09-17.

Additional context

I have a draft fix (adds a watchResyncTimeoutJitterFactor field and a new SetWatchResyncTimeoutJitterFactor option in gitops-engine, wired to a new ARGOCD_CLUSTER_CACHE_WATCH_RESYNC_JITTER_FACTOR env var in argocd-application-controller, default 0.1, 0 = disabled/backward compatible) and will open a PR referencing this issue.