Baike.dev
Log in
Back to tool/Back to issues
#4824·openreplay

Redis health probes leak timeout zombies on legacy installs; upgrades leave databases release unchanged

Author: andreafalzettiCreated Aug 11, 2026Updated Aug 12, 2026
Labelsdeployment

Describe the issue

The Redis health scripts vendored in the databases chart wrap redis-cli with GNU timeout:

bash
response=$(
  timeout -s 3 $1 \
  redis-cli -h localhost -p $REDIS_PORT ping
)

On an upgraded self-hosted installation whose databases release still uses docker.io/rjshrjndrn/redis:7, PID 1 is redis-server and there is no init/reaper. Terminated timeout processes accumulate as zombies with PPID 1.

Both local liveness and readiness probes run every 5 seconds. We measured approximately 24 new zombies/minute. When the container reaches pids.max=37552 after about 26.08 hours, probes fail with:

timeout: fork system call failed: Resource temporarily unavailable

Kubelet then restarts Redis. OpenReplay services that depend on it temporarily fail with connection refused.

This recurs deterministically; we observed 218 Redis restarts before isolating the cause.

There is also an upgrade-path gap: v1.27.0 defaults to ghcr.io/openreplay/valkey:8, whose amd64 image currently uses /sbin/tini -- /usr/bin/entrypoint.sh, but or_helm_upgrade() upgrades only toolings and openreplay. An existing databases release therefore remains on the legacy Redis image/chart. The v1.27.0 health template still contains the inner timeout.

Steps to reproduce the issue

  1. Use an existing OpenReplay installation with Helm release databases, Redis chart 12.10.1 and image docker.io/rjshrjndrn/redis:7.
  2. Confirm PID 1 is redis-server without an init/reaper.
  3. Leave the default local liveness/readiness exec probes enabled (both period 5s), or invoke those scripts repeatedly.
  4. Inspect zombie processes and the cgroup PID counter:
bash
kubectl -n db exec redis-master-0 -- ps -eo pid,ppid,stat,comm
kubectl -n db exec redis-master-0 -- cat /sys/fs/cgroup/pids.current
kubectl -n db exec redis-master-0 -- cat /sys/fs/cgroup/pids.max

The number of Z timeout children with PPID 1 and pids.current continue to grow.

  1. Run the documented v1.25 -> v1.27 release upgrade.
  2. Observe that the databases Helm release and Redis image are unchanged.

Expected behavior

  • Health probes must not leak processes or consume the container PID budget.
  • Existing installations should receive a safe, explicit migration path or at least a preflight warning when their cache image lacks an init/reaper and the databases release is not upgraded.
  • A release upgrade should make clear that changing the app release does not migrate the cache. A migration should be cache-scoped so it does not unexpectedly change PostgreSQL and MinIO/RustFS at the same time.

Proposed fix

  1. Remove the inner GNU timeout from all Redis/Valkey health-script variants and rely on Kubernetes timeoutSeconds for exec probes.
  2. Keep tini (or an equivalent reaper) in the supported cache image as defense in depth.
  3. Add a regression test that executes the probes repeatedly and asserts that zombie count and pids.current do not grow.
  4. Add an upgrader preflight for legacy cache images / stale databases releases.
  5. Provide a documented opt-in cache-only Redis -> Valkey migration that preserves Service, PVC, credentials and settings without implicitly upgrading PostgreSQL or MinIO/RustFS.

Proposed safe workaround (server-side dry-run validated)

The proposed patch changes only ping_liveness_local.sh and ping_readiness_local.sh to call redis-cli directly. Kubelet still enforces the outer probe limits (timeoutSeconds=6 liveness and 2 readiness in this installation). Because the ConfigMap is mounted as a projected volume rather than via subPath, the change propagates without restarting Redis.

The JSON patch has been validated against the live ConfigMap with kubectl patch --dry-run=server; it has not yet been applied to this production installation. Once applied, existing zombies will remain until Redis next restarts, so zombie/PID growth must be sampled before and after the change. The workaround should remain in place across the app upgrade until the cache is migrated to an image with a verified reaper.

OpenReplay Environment

  • OpenReplay version: v1.25.0, planning/pinning upgrade to v1.27.0
  • Tracker version: 17.2.x on v1.25; 18.x planned with v1.27
  • Kubernetes: k3s v1.31.5+k3s1, single node
  • Cache chart/image: Redis chart 12.10.1 / docker.io/rjshrjndrn/redis:7
  • Cloud provider: Hetzner Cloud
  • Observed cgroup PID limit: 37,552

Additional context

Relevant v1.27.0 sources:

Source: openreplay/openreplay

View original on GitHubView discussion on GitHub