OpenSnapshotBackend renames db without directory sync, which could cause data inconsistency after crash
Bug report criteria
- This bug report is not security related, security issues should be disclosed privately via the report form.
- This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
- You have read the etcd bug reporting guidelines.
- Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.
What happened?
When a member updates its db with one received from the leader, it renames ".snap.db" to "db" and the old database's final commit writes the leader snapshot's index into the old file, which does not contain those entries. Without running fsync on member/snap, if the member powers off just after the old database's consistent index is updated and before the rename reaches disk, the next time it starts, it would run on a database that contains a wrong consistent index.
What did you expect to happen?
We should fsync the directory right after renaming ".snap.db" to "db" in OpenSnapshotBackend.
How can we reproduce it (as minimally and precisely as possible)?
- Start a 3-member cluster
- Stop one follower and write 300 keys through the leader, so the leader sends the follower a snapshot.
- In the follower's member/snap directory, run "ln db db.old-inode" to keep the old db file reachable after the rename.
- Restart the follower and wait for "closed old backend file" in its log. Then kill etcd and simulate the lost rename with "ln db .snap.db && mv db.old-inode db".
- Restart follower. It runs on a database with wrong consistent index with it, missing the 300 keys.
Anything else we need to know?
Similar issue as #22340. #22340 points at the missing directory fsync where etcd receives a database snapshot. This issue points at the missing directory fsync where the snapshot is adopted as the live db.
Etcd version (please run commands below)
$ etcd --version
etcd Version: 3.7.2
Git SHA: 68c065e
Go Version: go1.26.8
Go OS/Arch: linux/amd64
</details>
### Etcd configuration (command line flags or environment variables)
<details>
Three-member cluster on one host (127.0.0.1). Default Settings. Linux 6.6 (WSL2), ext4 with data=ordered and commit=5.
</details>
### Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
<details>
</details>
### Relevant log output
```Shell
Source: etcd-io/etcd