[Bug]: Orphan-PVC restore deletes a restored Pod's stale init container too late, gate deadlocks on it first
Is there an existing issue already for this bug?
- I have searched for an existing issue, and could not find anything. I believe this is a new bug.
I have read the troubleshooting guide
- I have read the troubleshooting guide and I think this is a new bug.
I am running a supported version of CloudNativePG
- I have read the troubleshooting guide and I think this is a new bug.
Contact Details
No response
Version
trunk (main)
What version of Kubernetes are you using?
1.37
What is your Kubernetes environment?
Self-managed: kind (evaluation)
How did you install the operator?
YAML manifest
What happened?
When a backup/restore tool restores a Pod along with its PVC, the orphan-PVC restore path (reconcileRestoredCluster) deletes that Pod (ensureOrphanPodsAreDeleted), so a fresh one is created with the correct bootstrap role.
That deletion occurs after a gate (ensureClusterRestoreCanStart) that waits for all init containers in every existing Pod to finish. The gate exists because backup tools often add their own init container to copy the PVC data back, and CloudNativePG must wait for it to complete.
The current gate code cannot tell that container apart from CloudNativePG's own bootstrap-instance init container. bootstrap-instance already ran once, before the backup. Kubernetes only remembers that an init container ran on a Pod object, not that it already did its job on the volume. So the restored Pod reruns bootstrap-instance from scratch, on a volume that already has live, restored data.
What that rerun does depends on the role baked into it when the Pod was first created: a join that waits forever for a primary that has no running pod (the hang), or a rerun of initdb, a separate, more serious bug tracked on its own.
Result: the deletion step that should replace the stale Pod never runs, because it waits for that same stale Pod's init container to finish, which it never does.
Expected: restore completes, cluster reaches Ready.
Related: #11491 was the certificate-wait version of the same gate stalling on a stale init container. #11492 fixed that by writing the certificate status earlier. It does not fix the stale container: past the certificate wait, it moves into the join or initdb step above, where it now gets stuck (or worse). #11492 should be reverted: no real new Pod ever reaches this gate before the cluster is marked initialized, so it only changes which failure mode a restored stale Pod hits.
Cluster resource
Relevant log output
{"level":"info","ts":"2026-09-18T04:01:27.784710913Z","msg":"Waiting for server to be available","logging_pod":"cluster-example-2","connectionString":"host=cluster-example-rw user=streaming_replica port=5432 dbname=postgres connect_timeout=5 options='-c wal_sender_timeout=0s'"}
{"level":"info","ts":"2026-09-18T04:01:27.784710913Z","msg":"DB not available, will retry","logging_pod":"cluster-example-2","err":"failed to connect to `user=streaming_replica database=postgres`: 10.0.0.10:5432 (cluster-example-rw): dial error: dial tcp 10.0.0.10:5432: connect: connection refused"}
{"level":"info","ts":"2026-09-18T04:01:32.786519430Z","msg":"DB not available, will retry","logging_pod":"cluster-example-2","err":"failed to connect to `user=streaming_replica database=postgres`: 10.0.0.10:5432 (cluster-example-rw): dial error: dial tcp 10.0.0.10:5432: connect: connection refused"}
{"level":"info","ts":"2026-09-18T04:01:37.791193915Z","msg":"DB not available, will retry","logging_pod":"cluster-example-2","err":"failed to connect to `user=streaming_replica database=postgres`: 10.0.0.10:5432 (cluster-example-rw): dial error: dial tcp 10.0.0.10:5432: connect: connection refused"}
(repeats every 5s, never recovers)Code of Conduct
- I agree to follow this project's Code of Conduct
Source: cloudnative-pg/cloudnative-pg