Hash-based schedule spreading for CronWorkflow (Jenkins "H" style) to avoid thundering herd
Summary
Currently, CronWorkflow.spec.schedules only supports standard cron syntax. When many CronWorkflows share the same schedule (e.g. */5 * * * * or 0 * * * *), they all trigger at exactly the same instant, causing a spike in resource usage (thundering herd / cron stampede) on the workflow-controller and downstream systems.
Jenkins solves this with the H symbol in its cron syntax (e.g. H/5 * * * *, H H * * *). H is deterministically derived from a hash of the job's name, so each job gets a distinct, but stable and reproducible, offset within the given range — spreading execution over time without requiring users to manually stagger schedules.
Use Cases
- I have dozens of CronWorkflows that all run "every 2 hours" or "daily at midnight." Because they use identical cron expressions, they all fire simultaneously, causing CPU/API spikes on the cluster.
- I want to avoid manually hand-picking different minute/hour offsets for every CronWorkflow to desynchronize them, since this is tedious and doesn't stay consistent as CronWorkflows are added/removed/renamed.
- I want the same CronWorkflow to consistently resolve to the same offset across controller restarts (unlike pure random jitter), so behavior is predictable and debuggable.
Proposal
Support an H token in CronWorkflow.spec.schedules, following the same semantics as Jenkins:
Hin a field is replaced with a pseudo-random but deterministic value (derived from a hash of, e.g.,namespace/name), stable across controller restarts.- Support
H(start-end)to constrain the hashed value to a specific range (e.g.H(0-7) * * *). - Support
H/stepfor hashed offsets with intervals (e.g.H/15 * * * *).
Related discussions/issues
- https://github.com/argoproj/argo-workflows/discussions/6272 — prior discussion requesting the same capability (Discussion, not yet an Issue)
- argo-cd #14241 — similar problem (thundering herd on reconcile), solved with a random jitter parameter instead of hash-based offsets
Alternatives considered
- Manually staggering
schedulesvalues per CronWorkflow (works but doesn't scale / is fragile). - A global random jitter config — simpler to implement, but non-deterministic per restart and doesn't allow per-job range constraints the way Jenkins'
H(range)does.
Message from the maintainers:
Love this feature request? Give it a . We prioritise the proposals with the most .
Source: argoproj/argo-workflows