Snapshot reaping can (in rare cases) permanently prevent a node from restarting

Potential bug found on v10.2 via Antithesis, on a 4-node cluster (3 voters + 1 read-only) built from tip of master (e1b34edc) running inside Docker containers on Linux.

In rare cases, a node can write a snapshot-maintenance plan to disk that can never succeed. Once that happens the node exits on every subsequent start, and no amount of restarting recovers it. The bad plan survives reboots because it is stored in the node's data directory, which is the same mechanism that normally makes snapshot maintenance crash-safe.

Impact

Liveness: A node goes down and stays down. Restarting it does not help, and it never rejoins the cluster on its own.

The node exits during startup, before it begins serving. Any subsequent restart attempt sees it exit again immediately, in an unbounded loop. Nothing about the node's state improves between attempts, because the failure happens while reading data that is already on disk. Recovery would require manual intervention on the node's data directory.

The bug likely affects all v10.x releases (the root cause may have been added in 2b568e85d, see below).

Triggers

Context: rqlite periodically compacts its snapshots — folding accumulated changes into the base snapshot and discarding what is no longer needed (see snapshot/DESIGN.md). That compaction is planned first and subsequently executed, with the plan written to disk so an interrupted compaction can be resumed after a crash.

The bug needs all of the following to hold at the moment a compaction is planned:

• The most recent base snapshot has no newer incremental snapshots on top of it. This is the state a node is in immediately after receiving a snapshot from the leader, which happens when a node falls far enough behind that the leader can no longer send it individual log entries.
• That base snapshot still carries data to fold in, so the compaction has work to do and reaches its final step.
• There is at least one older snapshot to discard, so compaction runs at all.
• The clock reading taken when the compaction is planned matches the one taken when that snapshot was originally named — see Root Cause. In practice this means either less than a millisecond elapsed between the two, or the system clock was stepped backwards in between (for example by an NTP correction or a VM migration), since the reading used is wall-clock time and is not monotonic.

Once the plan is written, the node fails on the next restart and on every restart after that.

Symptom

While running, repeatedly:

reap failed: executing reap plan: rename <dir> <dir>: file exists

On every subsequent start, fatally:

failed to open store: failed to create snapshot store: executing reap plan: executing reap plan: rename <dir> <dir>: file exists

Note that the source and destination paths in the message are identical. While a POSIX rename of this kind would be a silent no-op, the Go standard library's os.Rename() has a different behavior of returning an error.

Sample logs

From the affected node. The snapshot directory name is identical on both sides of the rename throughout:

[snapshot-store] 2026/08/26 21:46:40 reap failed: executing reap plan: rename /rqlite/file/data/wsnapshots/17-1039-1787780800033 /rqlite/file/data/wsnapshots/17-1039-1787780800033: file exists
[rqlited]        2026/08/26 21:46:42 failed to open store: failed to create snapshot store: executing reap plan: executing reap plan: rename /rqlite/file/data/wsnapshots/17-1039-1787780800033 /rqlite/file/data/wsnapshots/17-1039-1787780800033: file exists
[snapshot-store] 2026/08/26 21:47:18 reap failed: executing reap plan: rename /rqlite/file/data/wsnapshots/17-1039-1787780800033 /rqlite/file/data/wsnapshots/17-1039-1787780800033: file exists
[snapshot-store] 2026/08/26 21:47:42 reap failed: executing reap plan: rename /rqlite/file/data/wsnapshots/17-1039-1787780800033 /rqlite/file/data/wsnapshots/17-1039-1787780800033: file exists
[rqlited]        2026/08/26 21:47:45 failed to open store: failed to create snapshot store: executing reap plan: executing reap plan: rename /rqlite/file/data/wsnapshots/17-1039-1787780800033 /rqlite/file/data/wsnapshots/17-1039-1787780800033: file exists

The failure recurs identically across restarts, minutes apart, on the same snapshot ID.

Root cause

A reap plan can contain a rename whose source and destination are the same directory. os.Rename fails with EEXIST when the destination is an existing directory — including when it is the source — and nothing in the plan, the executor or the resume path handles that case.

Where the two names come from. Snapshot directories are named by snapshotName(term, index) at: https://github.com/rqlite/rqlite/blob/e1b34edc2b572ed7ec02cac4030ebad57d57cb1f/snapshot/store.go#L960-L964

Thus function is called in exactly two places for this bug to trigger. Store.Create uses it to name a snapshot when one is created, and Store.reapInternal uses it again at the final step of a compaction to name the result:

https://github.com/rqlite/rqlite/blob/e1b34edc2b572ed7ec02cac4030ebad57d57cb1f/snapshot/store.go#L620-L628

Why they can be equal. When there is no incremental snapshot newer than the base one, newest resolves to full — the very snapshot being renamed. newID is then rebuilt from that snapshot's own term and index, so it agrees with the existing directory name in every field except the trailing timestamp. There are two ways that the timestamp component msec can also be equal:

1. Less than a millisecond elapsed between naming the snapshot and planning the compaction. The value is a millisecond quotient, so the two nanosecond readings differ while the quotient does not.
2. The clock moved backwards. UnixNano is wall-clock time and carries no monotonic guarantee, so an NTP step or a VM migration can make a later call return an earlier reading. This requires no sub-millisecond timing at all, and can reproduce the earlier value across an arbitrarily long interval.

AddRename then appends a rename with Src == Dst (persisted to the plan),

Why it creates a permanent issue.

• Executor.Rename is documented as idempotent, but only for the case where the source is missing and the destination is present. It inspects os.IsNotExist, and EEXIST is not that, so the error propagates.

  https://github.com/rqlite/rqlite/blob/e1b34edc2b572ed7ec02cac4030ebad57d57cb1f/snapshot/plan/executor.go#L27-L41)

• Checker.RenameDone, used by check() to detect a plan that already ran to completion, returns false while the source exists. With Src == Dst the source always exists, so the plan is never recognized as complete. https://github.com/rqlite/rqlite/blob/e1b34edc2b572ed7ec02cac4030ebad57d57cb1f/snapshot/plan/checker.go#L25-L37

• The plan file is removed only after a successful execution, so it is never removed.

Since the source is also the destination, it exists on every attempt, and the plan fails identically forever. NewStore resumes the persisted plan through check() before the node starts serving, so the failure is fatal at startup rather than merely noisy at runtime.

Potential fix

In snapshot/store.go, do not plan a rename that is a no-op:

 		// 7. Rename to new snapshot name. The end result of the Reaping process
 		// will be a new full snapshot with a new ID. That ID is generated from
 		// the newest snapshot's index and term, and the current timestamp.
+		// Skip the rename if it would be a no-op.
 		finalDir := filepath.Join(s.dir, newID)
-		p.AddRename(full.path, finalDir)
+		if full.path != finalDir {
+			p.AddRename(full.path, finalDir)
+		}

This prevents the bad plan from ever being written.

Repro / Validation

In my subsequent testing with Antithesis, the above fix made the failure go away. I don't know if this is the logically correct solution though. Happy to validate any other fix.

It is also possible to test this locally with a unit test using synctest to freeze the clock between snapshotName calls.