Data divergence (Safety violation) due to Async Heartbeat

Hello,

When testing this library with Antithesis, we encountered a serious data-divergence bug where some nodes on the Raft cluster commit different values to the FSM as compared to other nodes. On further investigation, we found that the bug violates at least three critical safety properties from Figure 3 of the Raft paper:

• Log Matching ❌
• Leader Completeness ❌
• State Machine Safety ❌

In some cases, the same bug can manifest in a different way, violating another crucial property:

• Election Safety ❌

Version tested: Latest main committed on May 19, 2026 (4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6)

Impact

Safety / Date Integrity - Cluster nodes can disagree on log entries with the same index/term, the values of committed entries at a given index, the committed FSM state, and in some cases even who the leader is for a given term.

Triggers

1. A network partition that causes a new leader to be elected in the majority partition.
2. A network restore that causes the old leader to receive heartbeats from the new leader
3. Concurrently, the old leader to be handling RPCs such as AppendEntries (e.g., servicing a client request) or RequestVote (e.g., another node timing out)

Symptom

Nodes disagree about the FSM value after some index.

For example, consider a 3-node cluster with a "chain-of-blocks" workload; that is, the FSM simply stores a running hash of all input commands ("blocks" of random data) that are applied to it. One sample symptom from the client's view is as follows:

Applying random blocks in sequence up to index 89, everything is fine:

[node1]: 2026/05/11 19:41:53 Applied block 89 (752358328303c56397112eac5446f69e) state: f7c73ec4 => cc97c2fc
[node3]: 2026/05/11 19:41:54 Applied block 89 (752358328303c56397112eac5446f69e) state: f7c73ec4 => cc97c2fc
[node2]: 2026/05/11 19:41:54 Applied block 89 (752358328303c56397112eac5446f69e) state: f7c73ec4 => cc97c2fc

At block 90, nodes diverge:

[node3]: 2026/05/11 19:42:01 Applied block 90 (2af01d1606f25b6237a659a02bdd8c1f) state: cc97c2fc => c45dc6f9
[node1]: 2026/05/11 19:42:02 Applied block 90 (270c66a1b96b16b17e0bbc2b4deb004b) state: cc97c2fc => 8dbda39d
[node2]: 2026/05/11 19:42:02 Applied block 90 (270c66a1b96b16b17e0bbc2b4deb004b) state: cc97c2fc => 8dbda39d

Reproducing

The bug is hard to reproduce exactly without Antithesis, but we have encountered it multiple times and can deterministically replay it once found. The general approach we followed in our test environment was as follows:

• State machine: Simple chain-of-blocks (or counter FSM) maintaining a single value as hash (or sum) of all inputs
• Cluster: 3 nodes
• Client: Random workload of Apply commands
• Fault Injection: Network partitions, node restarts
• Test oracles: Check for Raft safety invariants from the paper at all times + check liveness of commit progress / FSM convergence when fault injection is turned off.

Here is a sample test report we see:

(This issue is only about the three failing safety properties, I am filing separate issues about the failing liveness properties since they have different root causes)

Root Cause

• The Raft paper assumes that all the operations (handling RPCs, client requests, elections, etc.) are atomic and don't interleave with each other. Essentialy, every node in the protocol is a finite-state machine.
• In Hashicorp raft, most state-changing operations are handled by a big switch-loop in the main thread, with several other floating goroutines doing async work that should not affect protocol state.
• However, incoming heart-beat messages (i.e., AppendEntries without an entry) are handled on the I/O "transport" thread itself, instead of queuing it up for the main thread's loop like with other RPCs (this is noted in the Divergence Docs).
• ‼️ CAVEAT is that an incoming heart-beat message (like any other AppendEntries RPC) can change the state if it carries a new term when someone else was elected leader, and this bumps up the global currentTerm which everything else in the main thread relies on, and also sets the current state to be FOLLOWER. It does not appear to be sound to perform these state changes concurrently with the main thread.

Here's how the bug causes divergence in a 3-node setup A/B/C:

1. Assume the network link between node A and node B is down.
2. Node A becomes candidate for term T, requests votes (eventually gets it from Node C)
3. Node B also runs for term T but no votes
4. Node B becomes candidate for term T+1, requests votes (gets it from Node C)
5. Node B wins election for term T+1
6. Node A wins election for term T (just received the old vote from Node C)
7. Node B sends out heart-beat AppendEntries with term=T+1, which Node A doesn't immediately receive
8. Node A gets a client request to apply a new data entry X, and it still thinks it is a leader for term T, so on the main thread it prepares to create a log entry (data=X, term=T) and send out AppendEntries to peers.
   • :heartbeat: HOWEVER: Just before it can create the log entry, the heart-beat from Node B sent in step 7 above reaches, setting currentTerm=T+1. Because this is done on the fast-path on the network-transport thread, it races with the main thread. :checkered_flag:
   • The main thread ends up preparing a log entry (data=X, term=T+1) :bangbang: and also dispatching AppendEntries with these values before the next main-loop iteration where it realizes it is actually now a follower for term T+1.
9. Things get really bad from here. Some nodes apply this bogus data=X, term=T+1 to their logs, while others follow whatever Node B (the true leader for term T+1) says, e.g., they might apply data=Y at the same index with term=T+1. Logs diverge in data but get committed since all say term T+1 for the same index.

Source code

This part of the source code executes on the transport thread when an incoming heart-beat message carries a higher term: https://github.com/hashicorp/raft/blob/4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6/raft.go#L1478-L1485

Since it races with the main thread, it can run concurrently with other AppendEntries or RequestVote handlers.

Alternate Manifestation

In some test runs, we have also observed this bug to violate Election Safety.

In such runs, the raft logs show different nodes winning elections for the same term:

  20:11:59.941133 [DEBUG] node-1: vote granted: from="node-2" tally=2 term=286
  [...]
  20:11:59.941133 [INFO]  node-1: election won: tally=2 term=286
  [...]
  20:11:59.944273 [DEBUG] node-0: vote granted: from="node-2" tally=2 term=286
  [...]
  20:11:59.944274 [INFO]  node-0: election won: tally=2 term=286

I believe this happens when an incoming Async Heartbeat (with higher term T1) races with an incoming RequestVote RPC (with an even higher term T2), causing a node to grant a vote for the T2 RequestVote and then setting its term to the lower T1; later it increments its term back up to T2 granting a possibly different vote.