Use a kill switch inside the checker when your uptime probes start amplifying an incident — one feature flag, read on every tick, that can disable the noisy checks and stop the retries at the source.
Reach for tuned backoff and jitter instead when the retry storm stays inside a single process and never fans out onto a dependency somebody else is paging for.
Both are cheap to build.
Only one of them lets you quiet a polling client while its target is already on fire.
I run cron and queue infrastructure, so most of my pages arrive as either "the job didn't run" or "the job ran four times." Health checking sits in the same family of problems: a small, frequent, automated request that multiplies badly when something upstream changes shape.
What follows is the runbook I settled on after a fleet of pollers turned a non-incident into a real one — the failure mode, where the switch belongs, the implementation, and how to verify the flip before you walk away from the terminal.
What actually turns a polling client's uptime checks into a retry storm?
Amplification.
A single check is one request every 15 or 30 seconds, which nobody notices; a fleet of checks with retries layered on top is a synchronized load generator pointed at whatever you decided was important enough to monitor.
The math is unkind.
Take 40 instances, a 5s interval, and 3 retries per failed attempt, and a dependency that normally handles a trickle of health traffic suddenly sees a couple thousand requests a minute — all of them arriving at the exact moment it's least able to absorb them.
Retries stack on top of the polling interval rather than replacing it, and because every poller sees the same failure at the same time, they all back off together and return together.
The Google SRE book calls out this shape under cascading failures, and the load pattern that comes out of it looks nothing like organic traffic: sawtooth spikes, perfectly aligned, growing until something sheds load.
The worst one I've dealt with wasn't even a real outage.
Our checker read off the top level of a health payload, a routine change moved it under , and the field I assumed was there came back as an empty string.
Comparison against never matched, so every probe in the fleet concluded the service was sick.
The log line we emitted said and nothing else — no field name, no body snippet, no status code. 38 pollers, 5s interval, 3 retries each, roughly 1,900 requests a minute against an endpoint that normally sees
120.
The service was healthy the entire time.
It took us about 25 minutes to work out that the fault was in the shape of the data, not the state of the system, and most of that time went into arguing about which deploy had "broken prod." I'm still not sure why we never logged the raw body on a parse miss; we do now.
That's the case a kill switch is for.
Not for hiding a real failure, but for cutting the amplifier when your own tooling has become the loudest client on the box.
Put the kill switch where the check runs, not where it's configured The instinct is to disable checks in the monitoring configuration — comment out the check, redeploy the config, move on.
Mid-incident that's the wrong layer.
Config lives on the deploy path, and the deploy path is exactly what you don't want to exercise while you're troubleshooting, because it's slow, it needs review, and it's a second source of change during an event you're already trying to explain.
The switch belongs in the polling client, evaluated on each tick, with the last known value cached in memory.
Where the switch lives Time to take effect Blast radius Usable mid-incident Config file plus redeploy minutes to hours whole service no Env var plus rolling restart minutes one deployment rarely Feature flag read per tick one interval one check, one cohort yes Circuit breaker in the client milliseconds one dependency automatic, not aimable A circuit breaker and a flag solve different halves of this.
The breaker reacts in milliseconds to error rate