Slack and MS Teams notifications are dropped when the provider returns HTTP 429

Bug description

SigNoz does not retry a Slack or MS Teams notification when the provider replies with HTTP 429 (rate limited). It treats the 429 as an unrecoverable error and stops the notification pipeline after one attempt.

Both providers use 429 to signal a temporary limit:

• Slack replies with 429 Too Many Requests and a Retry-After header on all its HTTP APIs, incoming webhooks included.
• The MS Teams notifier targets Power Automate Workflows. A workflow endpoint replies with a real HTTP 429 when it is throttled.

Other notifiers in SigNoz already retry after a 429 response: Google Chat, incident.io, Jira, Opsgenie and PagerDuty. Only Google Chat is SigNoz's own code, the rest inherit the behavior from upstream alertmanager, and PagerDuty sets 429 on its Events v2 branch only.

Impact

Because the pipeline stops, SigNoz does not record the notification as sent. Two results are possible:

• If the alert still fires at the next flush, the notification is late by as much as 5 minutes, the default group_interval. A 5xx response instead retries with backoff and usually recovers in seconds.
• If no alert in the group still fires at the next flush, SigNoz sends no notification at all. The user gets no firing message and no resolved message. The default route groups by alertname only, so a sibling series that still fires masks the loss.

The second result is a silent loss of the alert. Rate limits occur during alert storms, which is when delivery is most necessary. Short alerts in a storm are the most probable to be lost.

SigNoz also writes an unrecoverable error to the log. A 429 response is recoverable.

Expected behavior

Slack and MS Teams retry after a 429 response, as the sibling HTTP notifiers do.

How to reproduce

1. Configure a Slack channel or an MS Teams channel for alerts.
2. Point it at an endpoint that replies with HTTP 429.
3. Cause an alert to fire.
4. Read the logs. They show notify retry canceled due to unrecoverable error after 1 attempts. SigNoz makes no second attempt.

Version information

• Signoz version: v0.142.0