[Bug]: Restoring a Pod along with its PVC can let a stale one-shot init container overwrite the data directory
Is there an existing issue already for this bug?
- I have searched for an existing issue, and could not find anything. I believe this is a new bug.
I have read the troubleshooting guide
- I have read the troubleshooting guide and I think this is a new bug.
I am running a supported version of CloudNativePG
- I have read the troubleshooting guide and I think this is a new bug.
Contact Details
No response
Version
trunk (main)
What version of Kubernetes are you using?
1.37
What is your Kubernetes environment?
Self-managed: kind (evaluation)
How did you install the operator?
YAML manifest
What happened?
The bootstrap-instance init container runs once per instance to initialize the data directory with initdb, pg_basebackup, or restore, whichever role the controller picked when the Pod was first created.
Kubernetes only tracks whether an init container ran on a Pod object, not whether it already did its job on the volume. So when a backup/restore tool restores a Pod together with its PVC, the new Pod reruns bootstrap-instance on a volume that already holds a database.
If that role is initdb, EnsureTargetDirectoriesDoNotExist tries to guard against this: it renames the existing data directory aside if pg_controldata can read it, or deletes it otherwise. Either way, the running instance loses the restored data: renamed data sits on disk under a different path, and deleted data is gone.
Reproduced locally: data written before the backup is no longer in the cluster after the restore, and the instance comes up on a fresh, empty data directory.
Expected: initdb never runs against a volume that already has data.
Related: #11503 is the join-variant of the same bug.
Cluster resource
Relevant log output
Code of Conduct
- I agree to follow this project's Code of Conduct
Source: cloudnative-pg/cloudnative-pg