Sustained Redis outage self-destructs multi-node cluster: health staleness fails K8s liveness on every node at once, then graceful drain hangs forever with no path back to SERVING

Summary

A ~7-minute outage of a single Redis instance took our entire 6-node LiveKit deployment from healthy to hard-down for ~53 minutes, and the cluster could not recover on its own — every node ended up SIGTERM'd by its Kubernetes liveness probe, stuck in an unbounded graceful drain, and re-registered in Redis as SHUTTING_DOWN with no path back to SERVING. Recovery required manually deleting all pods.

We believe this decomposes into three interlocking behaviors (details below): (1) the / health endpoint transitively measures Redis pub/sub liveness, (2) the default helm chart uses that same endpoint for both liveness and readiness (livekit-helm#116), and (3) Stop(force=false) waits for participants forever with no configurable deadline (acknowledged as by-design in #1032) and a drained node can never return to service.

Environment

• livekit-server v1.11.0 (livekit/livekit-server:v1.11.0), 6 replicas, EKS, hostNetwork (media over direct UDP to each node's public IP)
• Deployed via livekit-helm chart 1.9.0 with its default probes: liveness = readiness = GET / (period 10s, failureThreshold 3)
• terminationGracePeriodSeconds: 18000
• Redis: single in-cluster instance (redis.inference.svc:6379); clients: JS SDK (browsers) + Rust SDK (server-side publishers)
• We diffed the relevant code paths (pkg/service/server.go healthCheck/Stop, pkg/routing/redisrouter.go) between v1.11.0 and v1.13.3 — byte-identical, so the latest release has the same behavior.

Timeline (UTC, 2026-07-11)

1. 09:07:00 — the k8s node running Redis is replaced; Redis is unreachable for ~7 min and comes back empty. livekit logs redis: connection pool: failed to dial ... connect: connection refused, then could not delete room / could not delete participant / could not store room for in-flight cleanups.
2. 09:07:0x — node stats stop updating on every livekit node at once: the keepalive ping that refreshes Stats.UpdatedAt is transported over Redis pub/sub (redisrouter.go statsWorker → PublishPing; keepaliveWorker → UpdateNodeStats). GET / starts returning 406 (healthCheck: 406 when Stats.UpdatedAt is >4s stale).
3. 09:07:53–09:07:59 — kubelet liveness (3×10s on GET /) SIGTERMs all 6 pods within 6 seconds: each logs exit requested, shutting down followed by ERROR failed to mark as draining: could not register node: dial tcp ...:6379: connect: connection refused (RedisRouter.Drain() — Redis still down).
4. 09:08 → 10:17 — every pod loops waiting for participants to exit every 5s, for 63–70 minutes each (756–835 iterations/pod). Stop(force=false) has no timeout; participants whose cleanup writes failed while Redis was down stay in roomManager indefinitely, and long-lived publishers kept streaming over direct UDP (unaffected by endpoint removal). The only ceiling was SIGKILL at 18000s.
5. ~09:11 — Redis recovers (empty). Each wedged node's resumed keepalive re-registers it with State=SHUTTING_DOWN (set earlier by Drain()), and the node selector requires NodeState_SERVING (selector/utils.go), so the cluster stays at zero schedulable nodes — while GET / returns 200 again (it only checks stats freshness, not drain state), so Kubernetes considers the pods healthy and routes signal traffic to them. Every room create/join returns HTTP 500 (we measured 17,301 signal-WS 500s + 4,842 room-connect failures over the window).
6. 10:00 — manual kubectl delete pod of all livekit-server pods restores service in ~2 minutes.

Total: a ~7-minute Redis blip → ~53 minutes of 100% signaling downtime with no self-recovery path.

A datapoint bounding the trigger

The next day the same deployment's Redis in another region was OOMKilled and restarted in-place within seconds (same pod IP, state also lost): zero livekit pods were affected — no liveness failures, cluster fully recovered on its own. So the failure mode is specifically sustained Redis unavailability (roughly >35–55s: 4s staleness threshold + 3×10s probe), not Redis restarts or data loss per se.

Expected behavior

1. A Redis outage should degrade routing, not fail node liveness: health used for liveness should reflect process health rather than Redis-pub/sub keepalive freshness — or the server should expose separate liveness/readiness endpoints (cf. livekit-helm#116, where the probe split is blocked on exactly this).
2. Graceful drain should be boundable: a drain_timeout config (or wiring DrainConnections(force) from #4618 — which we noticed is merged but has no call site in the OSS Stop() path) so participants that cannot be cleaned up don't hold the process hostage forever.
3. A node whose drain was triggered spuriously (here: by the liveness kill itself) should have some way back to SERVING, or at minimum this Redis↔health coupling should be documented prominently for self-hosted Kubernetes deployments, since the default chart configuration turns any >~1 min Redis outage into a full-cluster kill.

Actual behavior

One non-HA Redis interruption → all SFU nodes killed simultaneously by their liveness probes → all stuck in waiting for participants to exit → permanent zero-SERVING-nodes state until manual force-restart.

Workarounds we're applying (for other operators)

• Decouple/relax the k8s liveness probe from GET / (requires patching the chart's hardcoded probes; see livekit-helm#116 / PR #161)
• Bound the blast radius with a probe-level terminationGracePeriodSeconds well below the pod-level one
• Alert on the failed to mark as draining / waiting for participants to exit log signatures
• Harden Redis placement so sustained outages can't happen (the short-blip case is handled fine by the server)

Happy to provide more logs/timestamps or test patches.