Deadlock / cluster stall after Leadership Transfer
Hello,
When testing this library with Antithesis, we encountered a bug where a healthy cluster is unable to make progress after a leadership transfer operation that causes a goroutine to get stuck.
Version tested: 4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6
Impact
Liveness - A healthy cluster is unable to commit new entries unless the affected leader is rebooted.
Triggers
- A leadership transfer from some node A to node B, when B's logs are not fully in sync, and the transfer is interrupted by node A having to step-down as a leader for some other reason (e.g., lease timeout).
- A subsequent re-election of node A as the leader.
Symptom
The re-elected leader (node A in the above example) refuses to service client requests with the error message "leadership transfer in progress"
Reproducing
The bug is hard to reproduce exactly without Antithesis, but the general approach we followed in our test environment was as follows:
- State machine: Simple chain-of-blocks (the FSM maintains a running hash of all input commands received)
- Cluster: 3 nodes, no membership changes
- Client: Random workload of Apply commands intermixed with occasional Leadership Transfer commands.
- Fault Injection: Network partitions, node restarts
- Test oracles: Check for Raft safety invariants from the paper at all times + check liveness of commit progress / FSM convergence when fault injection is turned off.
Sample Logs from a 3-node test
Most logs are from raft, whereas the last line below is from an Antithesis eventually test command that checks for the cluster's ability to make commits when fault injection is stopped and the cluster is fully reachable via the network.
[...]
2026-07-02T00:08:27.504Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950907504.tmp
2026-07-02T00:08:27.504Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950885935
2026-07-02T00:08:27.504Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:27.504Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:08:32.868Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:08:36.448Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:08:36.448Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950916448.tmp
2026-07-02T00:08:36.449Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950894124
2026-07-02T00:08:36.449Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:36.449Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:08:39.537Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:08:43.792Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:08:43.792Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950923792.tmp
2026-07-02T00:08:43.792Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950902333
2026-07-02T00:08:43.792Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:43.792Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:08:49.300Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:08:52.752Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:08:52.752Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950932752.tmp
2026-07-02T00:08:52.752Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950907504
2026-07-02T00:08:52.752Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:52.752Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:08:57.359Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:08:59.551Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:08:59.551Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950939551.tmp
2026-07-02T00:08:59.551Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950916448
2026-07-02T00:08:59.551Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:59.551Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:09:02.553Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:09:07.312Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:09:07.312Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950947312.tmp
2026-07-02T00:09:07.312Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950923792
2026-07-02T00:09:07.312Z [INFO] node-0: no logs to truncate
2026-07-02T00:09:07.312Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:09:07.630Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:09:10.859Z [ERROR] eventually-workload-commit-progress: commit-progress give-up; cluster healthy but no fresh entry committed: attempts_taken=24 baseline_applied_index=147 last_err="apply rejected: leadership transfer in progress" per_node="map[raft-node-0:8400:map[applied_index:138 last_index:143 leader_id:node-2 reachable:true state:Follower term:169] raft-node-1:8400:map[applied_index:138 last_index:147 leader_id:node-2 reachable:true state:Follower term:169] raft-node-2:8400:map[applied_index:147 last_index:147 leader_id:node-2 reachable:true state:Leader term:169]]"Root Cause
The root cause is a deadlock during a leadership transfer operation that does not immediately signal any warnings but bites you way into the future by getting the cluster stuck.
- When you initiate a leadership transfer from A to B, the current leader A first tries to get B's logs up to date via an async replication thread.
- The leadership transfer logic is a separate goroutine that waits for this replication before telling B to takeover.
- While all this is happening, A might have to step-down as leader for unrelated reasons (e.g., the lease timer runs out, or it cannot reach a quorum due to network) and say another node C beomes leader.
- When A steps-down as leader, it aborts all replication threads (including the A->B catchup) but the leadership transfer goroutine is still blocked waiting for it to complete (an implementation bug).
- There is no problem so far, because C is the new leader and it does its job for a while.
- Unfortunately, the leadership transfer goroutine in A has set a global flag "leadership transfer in progress" and this flag cannot be unset until it is unblocked.... but it never will be!!!
- Crucially, this flag does not prevent it from participating in elections and the flag does not get reset if it wins a future election.
- So in the future, A can legitimately get relected as leader but it will refuse to accept any client requests and stall on future commits with the reason "leadership transfer in progress".
- When the network is healthy and A is a leader, there is no way to get out of this other than rebooting the node A to clear the flag.
Source code
When the leadership transfer is interrupted with the replication torn down, the transfer goroutine is stuck forever (deadlocked) here: https://github.com/hashicorp/raft/blob/4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6/raft.go#L1003
The leadership transfer flag is only reset here, but this defer never runs because nothing is sent on doneCh:
https://github.com/hashicorp/raft/blob/4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6/raft.go#L727
Because of this, when the node gets re-elected as leader in the future, it will always error out, for example: https://github.com/hashicorp/raft/blob/4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6/raft.go#L908-L910
Related Issues
I believe this is different from #498 since the deadlock occurs at a different program location.
Our testing has also revealed safety-property violations which I have reported in #695.
Source: hashicorp/raft