#10788·tldraw

Deploy new Grafana alert rules without a manual group bootstrap

Author: drievCreated Sep 15, 2026Updated Sep 15, 2026
Labelsinternalautomations

️: Ah, open an issue to fix this properly, so future alerts don't break the step.


The Grafana resources deploy fails at its alert-rules step whenever a brand-new alert rule reaches main: Grafana refuses to create a rule that already carries group labels, and the PR dry run cannot catch this because alerting has no server-side dry run, so the rule reports as "skipped" and the PR goes green. The documented workaround is a manual, one-off bootstrap of the rule group before merging, and it has now been missed on both rules added this way (PR #10052 and PR #10695), leaving the alerts unprovisioned until someone notices the red deploy. The ask is for the deploy to handle a new alert rule on its own, so that adding a rule file to the tree is enough and the step stays green; the existing dashboards should keep deploying independently of alert-rule failures, as they do today.

Open questions

  1. Should the deploy create the missing group itself on the push, or should the PR check fail early when a rule names a group that does not exist yet, or both? Awaiting answer.
  2. Should the two version chain rules from #10695, which are still unprovisioned, be bootstrapped by hand now or left as the first test of this fix? Awaiting answer.

Confidence: 78%, ready to get started.