Service-scoped alarm rules

Background

Pinpoint is introducing service concept: service-module owns the service table(uid, name), ServiceUid is threaded through the application index and the web layer, and applications and permissions is being organized under a service.

This issue and following PRs will be implementing alarms under this service concept.

Problems with the current model

Beyond the missing service dimension, the schema limits what a rule can express:

1. One checker, one threshold per row. A rule is a single CheckerCategory value plus one threshold int. Expressing "error rate above X and slow count above Y over a 5 minute window" means several unrelated rules.
2. Notification is inconsistent between transports. Email and SMS are boolean columns on the rule (sms_send / email_send) delivering to the single user_group_id that rule names, so one rule cannot notify two groups or route transports to different recipients. Webhooks, by contrast, are addressable rows joined many-to-many through webhook_send — the shape email and SMS lack, bolted on separately. The flag and the mapping can also disagree: webhook_send = '1' with no matching webhook_send row is representable and silently sends nothing.
3. No per-rule interval or state. alarm_rule has no interval or last-checked column, so every rule shares one cron period and is evaluated on every cycle; a rule cannot be checked hourly while another is checked every minute.
4. No rule reuse. Applying the same set of alarms to twenty applications means creating them twenty times, and changing them means editing twenty.

Defect: deleting an application leaves its alarm rules behind

This one is independent of the feature work and reproducible today.

ApplicationCleanupTasklet removes inactive applications from the application index. Not evaluating a rule whose application is gone is correct — and that is what happens, since AlarmReader.fetchApplications() only yields applications present in the index.

The defect is that nothing ever removes those alarm_rule rows.

They are not reachable in the UI either: the alarm list is fetched by applicationId and a deleted application can no longer be selected, so nobody sees them to clean them up. The rows simply accumulate, permanently inert, and the count grows every time the cleanup job runs.

Proposal

Introduce a service-aware alarm model alongside the existing one, and leave alarm_rule in place for existing users.

Reusable rule sets. A template (a named set of rule definitions) that can be applied to one or more applications, with per-rule overrides, so a service owner defines a policy once.

Condition trees. A rule holds a condition tree (AND/OR over leaves, each leaf a metric with an aggregation, window and comparison) instead of one checker plus one threshold.

Notification channels as rows. Channels (email / SMS / webhook) become addressable rows bound to rules, replacing the boolean flags, so a channel is defined once, reused, and can be reported on — the shape webhooks already have, applied uniformly. A rule with no channel bound is a valid configuration: it records history without notifying, which matches how CloudWatch alarms with no actions and Datadog monitors with no recipients behave. It should, however, be visually distinct from a rule whose channel exists and has no reachable destination.

Due-based evaluation with an outbox. Each rule carries its own check interval and state (status, last checked, next check time). A periodic tick still drives the batch, but it selects the rules whose next check time has passed instead of walking the application index, so intervals become per-rule and evaluation no longer depends on the application being indexed. Firing writes a history row and enqueues deliveries into an outbox table in the same transaction; a separate dispatcher drains the outbox, so notification failure cannot poison the evaluation transaction.

Pluggable metric sources. The metric query behind a condition is resolved per data source, so new metric backends can be added without touching the evaluation loop.

Explicit application-existence check. Walking the application index gave evaluation an incidental skip for applications that no longer exist. Selecting rules by due time loses that, so the check has to become explicit — otherwise a rule whose application was deleted keeps querying its metric backend on every interval, which is correct in outcome but wasteful and noisy.

It is worth a small interface rather than a direct ApplicationDao call. A rule identifies its target by service name, while ApplicationDao takes a ServiceUid, and the conversion lives in ServiceUidService in the web module — which the alarm code should not have to depend on.

Cascade application deletion. Extend the cleanup job that deletes an application so it also removes that application's alarm rules and their dependent rows, in the same job. An event-time hook is preferable to a periodic sweeper: a sweeper has to infer "dead" from indirect signals, and would mass-delete after any extended outage of the alarm batch.

Migration

No backfill. Existing alarm_rule rows keep working against the existing pages (with DEFAULT service); users re-create the alarms they still want under the new model as they adopt services. Given the accumulation described above, a straight copy would carry inert rules into the new tables.