Active-healthcheck targets stuck UNHEALTHY with no auto-recovery on 3.9 dbless (5xx ejection not recovering; orphan status freezes counter)

Summary

On Kong 3.9 / dbless, active-health-checked upstream targets get stuck UNHEALTHY and are not re-marked healthy automatically once the backend recovers. (Our upstreams are produced by KIC via the konghq.com/upstream-policy annotation + KongUpstreamPolicy CRD, but this appears to be a core health-check behavior, not tied to that path — see note under Steps.) We have observed this from two distinct triggers, and in dbless there is no surgical recovery — only a full kong reload / pod restart clears it (POST /upstreams/{u}/targets/{t}/healthy returns HTTP 500 in dbless; targets are declarative).

Trigger A — normal unhealthy ejection (5xx) does not auto-recover in production

A target ejected by a listed unhealthy status (e.g. 503) is correctly marked down, but after the backend returns to 200 the active check does not bring it back — it stays UNHEALTHY until a restart. This is the behavior class of #5531 / #7566 (closed as stale), still present on 3.9 dbless. It is our most frequent trigger in production (e.g. after a DB/Redis failover the backend 503s briefly, is ejected, recovers to 200, but the target stays stuck).

Trigger B — "orphan" status (not in either list) freezes the counter

When an active probe returns a status in neither healthy.http_statuses nor unhealthy.http_statuses (e.g. 404), the counter increments in no direction and active-check logging for the target stops (consistent with #8324). Real-world case: an ingress route in front of the target was briefly absent (a 404) during a backend namespace migration → the target froze and did not recover.

Expected Behavior

• After a target is ejected (whether by a listed unhealthy status or an orphan status), active checks should keep running and re-mark it healthy once it returns a healthy status.
• An orphan status should not permanently block recovery (count it as unhealthy, or treat it as neutral).

Steps To Reproduce (Trigger B, deterministic)

Note: this is a Kong core health-check behavior (lua-resty-healthcheck), not specific to how the upstream is declared — the same happens with a raw declarative kong.yml upstream or an Admin-API-created one. We show the KIC form below because it is our setup; the upstream/healthchecks are produced from a KongUpstreamPolicy CRD referenced by the konghq.com/upstream-policy Service annotation.

apiVersion: configuration.konghq.com/v1beta1
kind: KongUpstreamPolicy
metadata:
  name: demo-policy
spec:
  algorithm: round-robin
  healthchecks:
    active:
      type: http
      httpPath: /health
      timeout: 5
      healthy:   { interval: 5, successes: 1, httpStatuses: [200, 302] }
      unhealthy: { interval: 5, httpFailures: 3, httpStatuses: [500, 502, 503] }
---
apiVersion: v1
kind: Service
metadata:
  name: demo-svc
  annotations:
    konghq.com/upstream-policy: demo-policy   # <-- upstream+healthchecks come from here
spec:
  ports: [{ port: 80 }]
  # backed by an EndpointSlice / target that serves /health

1. Run Kong 3.9 dbless + KIC with the above.
2. Backend /health returns 200 → target HEALTHY.
3. Make /health return 404 for a sustained period, then restore it to 200.
4. Observe: the target does not recover; active-check logs for it stop; only kong reload / restart restores it.

Trigger A (5xx non-recovery) is intermittent: in isolated/idle testing (Chaos Mesh: connection-refused, timeout, 5xx, packet loss, CPU throttling, load) the same setup self-heals within seconds, so we could not reproduce it on demand. It reproduces in production under real load, matching #5531. We report it because it is our dominant real-world symptom, even though it is not deterministically reproducible.

Context

• Kong version: 3.9 (OSS, kong:3.9), dbless. KIC 3.5, upstreams via konghq.com/upstream-policy + KongUpstreamPolicy CRD. Kubernetes (AKS). Cross-region upstream with two targets (one per region) + algorithm: latency.
• Related, closed as stale: #5531, #7566, #8324. Plugin/status interaction (fixed, passive): #10281.
• In dbless the documented recoveries (re-mark healthy / update target via Admin API) are unavailable, so a stuck target is a sustained per-target outage until the gateway is restarted.