What Kubernetes Health Checks Actually Guarantee

What Kubernetes Health Checks Actually Guarantee

2026年9月4日1 次浏览来源:Dev.to阅读原文

Kubernetes health checks tell you whether a workload satisfies the narrow condition the probe was designed to test.

They do not tell you whether the business objective that workload exists to serve actually happened.

That distinction sounds pedantic until it's the reason a deployment shows green across every dashboard while the checkout flow silently stopped completing orders forty minutes ago.

Passing health checks prove participation in orchestration.

They do not prove the intended outcome occurred.

What Kubernetes Actually Guarantees Strip away the operational folklore and a readiness or liveness probe answers a narrow question: did the process respond to this specific check, within this specific timeout, in this specific way?

A liveness probe confirms the process hasn't deadlocked or hung badly enough to fail its own restart threshold.

A readiness probe confirms the process is willing to accept traffic right now.

A startup probe confirms initialization completed before the kubelet starts evaluating the other two.

None of these checks execute business logic.

None of them touch a real code path a customer would traverse.

A readiness probe hitting and getting a 200 tells Kubernetes that the container is ready to receive traffic — it does not tell Kubernetes "I can complete a checkout," "I can authenticate a user," or "I can write a record that will still be readable tomorrow." The probe's contract is with the control plane, not with the business objective the control plane exists to serve.

This is the correct design, not a flaw.

Kubernetes has no way to know what your business objective is, and it would be architecturally wrong for the orchestrator to try to infer it.

The control plane's job is process lifecycle management: keep the right number of willing replicas in rotation, and route traffic only to the ones that said yes.

It does that job well.

The failure isn't in what Kubernetes guarantees — it's in what architects quietly assume it guarantees once the dashboard turns green, and closing that gap is squarely a cloud architecture strategy decision, not an implementation detail to delegate downward.

What Kubernetes Does Not Guarantee The gap between "process is participating" and "objective is being met" isn't unique to Kubernetes.

It shows up at every layer of the stack where a proxy signal stands in for direct measurement of the thing you actually care about.

Six of the most common false-positive patterns: Signal What It Confirms What Still Fails Underneath Readiness probe Process accepts a TCP connection or returns 200 on a health endpoint Downstream dependency (database, cache, upstream API) is unreachable; traffic routes to a pod that can't complete a real request DB connection check Pool has a live connection, returns Replication lag has pushed reads stale, or a lock is queuing writes indefinitely while the ping itself never touches the locked table LB target health check Target instance responds on the configured health path/port Application logic returns malformed data or a 5xx on the actual transaction path the health path never exercises Circuit breaker Breaker reports closed — failure rate hasn't crossed the trip threshold Individual transactions fail below the statistical threshold the breaker was tuned to catch; the breaker is doing its job correctly and that job doesn't cover this case Backup job Process exits 0 Backup is incomplete, unusable, or not restorable even though the job reported successful completion Auth/identity service Port is reachable, service responds to a liveness ping Token validation logic can fail open under certain error conditions, and a liveness ping never exercises those conditions Six control points, six different ways a passing signal hides a real failure.

The pattern repeats because every one of these checks was built to answer a cheap, fast, narrow question — is the endpoint alive — rather than an expensive, slow, broad one — did the operation actually accomplish what it was supposed to.

That tradeoff is usually correct.

The problem is treating the cheap question's answer as though it settled the expensive one.

The readiness-probe row above isn't hypothetical, either — the Network Loop in the Rack2Cloud Method's Day-2 operations guide documents the same failure mode from the routing side: a misconfigured readiness probe sending traffic to a pod that can't actually serve it.

Why Healthy Systems Still Fail Here's the scenario that makes this concrete.

A deployment rolls out.

Every readiness probe passes within its configured threshold.

The rollout controller reports success.

The dashboard shows 100% of replicas ready, zero restarts, latency within SLO.

By every signal Kubernetes was designed to emit, this deployment succeeded.

Forty minutes later, support tickets start arriving.

Checkout is broken — not down, broken.

Users click "place order," get a spinner, and the order never appears in the downstream fulfillment system.

The readiness probe on the checkout service never checked whether it could reach the fulfillment queue.

It checked whether the process was up.

The process was up.

The queue connection, established once at startup and never re-validated, had gone stale after a network policy change nobody connected to this deployment.

The severity isn't hypothetical.

The proxy can remain healthy while the business function is already broken.

Nothing in the orchestration layer was lying — the rollout genuinely succeeded by every metric Kubernetes tracks.

That's the uncomfortable part — this isn't a monitoring gap you close by adding more Kubernetes-native checks.

It's a category gap.

You can add ten more readiness probes and still never catch this, because readiness probes operate at the orchestration-participation layer and the failure lives at the business-outcome layer.

No amount of depth within the wrong category closes a gap between categories.

Contrast this against the inverse case: a probe that fires correctly and gets misdiagnosed as something else entirely.

Kubernetes Day-2 Incidents documents a CrashLoopBackOff traced back to an IAM permission gap, where exit code 0 on a crash loop means the readiness probe itself is failing — a different problem entirely, and one where the probe is doing exactly what it should.

That post is about misdiagnosing which layer causes a visible failure.

This one is about a signal correctly reporting success while the objective still fails.

Both are real; they're not the same failure.

The False Completion Pattern What just happened has a name, and it's not specific to Kubernetes.

This is False Completion — the failure mode in which an operation reports success by system metrics while the underlying objective was not actually met.

The pattern generalizes cleanly.

Swap the readiness probe for an ALB health check, a backup job's exit code, or a circuit breaker's closed state, and the shape of the failure is identical: a signal designed to answer a narrow, cheap question gets treated as authoritative for a broad, expensive one it was never built to answer.

The same pattern shows up applied to HTTP response codes — a 200 status reporting success while the actual transaction underneath it failed.

Kubernetes is the evidence layer in this post, not the root cause — the root cause is architectural, and it recurs everywhere a proxy metric substitutes for direct measurement of the objective.

False Completion tends to survive in production specifically because the proxy signal is healthy.

A system that fails loudly gets fixed.

A system that reports success while silently missing its objective can run for weeks.

The gap surfaces at incident time, when a human notices the business symptom and has to work backward through several layers of "but the dashboard was green" to find it.

That's the most expensive possible time to discover it, and it's discovered there specifically because nothing upstream was built to surface it earlier.

The architectural lesson isn't "distrust Kubernete

分享
Baike.dev

baike.dev helps you discover great languages, frameworks, databases, DevOps and cloud-native tools.

Quick links

About

Contribute

Found a great developer tool? Share it with the community.

Submit a tool
© 2026 baike.dev Developer EncyclopediaUpdated daily · Discover great developer tools