#711·raft

TestRaft_HasExistingState fails intermittently under go test -count>1

Author: rowhitswamiCreated Sep 8, 2026Updated Sep 8, 2026

TestRaft_HasExistingState fails intermittently when the package is run with -count>1. I could not reproduce it at -count=1 in 47 attempts across three scopes, so it looks specific to in-process repetition rather than something you would hit in normal CI.

Raising it because -count=N with -shuffle is the usual way to hunt genuine flakes, and this adds noise to that on the raft package. As far as I can tell it is not in #372's list and has no existing issue, PR or commit.

Failure

--- FAIL: TestRaft_HasExistingState (5.20s)
    testing.go:705: peer mismatch:
      {Servers:[{Voter server-b3bfde62 ...} {Voter server-c21909b3 ...}]}
      {Servers:[{Voter server-b3bfde62 ...} {Voter server-c21909b3 ...} {Voter server-1a02a2fe ...}]}

Two servers expected, three observed — a node from elsewhere in the run appears in the configuration.

What I measured

At b853494d, go1.25.4, darwin/arm64. Every execution a fresh process.

scope flags runs HasExistingState failed
this test alone -count=1 -shuffle 25 0
this test alone -count=3 -shuffle 25 0
root package -count=1 (no shuffle) 12 0
root package -count=1 -shuffle 12 0
./... -count=1 10 0
./... -count=3 -shuffle 31 2

It needs the full package and -count>1. Seeds for the two failures: 758137489003, 837114709011.

Reproduction

go test -count=3 -shuffle=758137489003 ./...

Roughly 1 run in 15, so expect to repeat it.

Other tests that failed in the same runs

All already known — listed only so the picture is complete, and because the rates may be useful:

test -count=3 runs note
TestRaft_FollowerRemovalNoElection 4/31 #669 open, unmerged
TestRaft_SendSnapshotFollower 2/31 #311, and in #372
TestRaft_ProtocolVersion_Upgrade_2_3 2/31 in #372
TestRaft_RecoverCluster 2/31 in #372
TestRaft_ProtocolVersion_Upgrade_1_2 1/31 in #372
TestNetworkTransport_AppendEntriesPipeline_CloseStreams 1/31 has prior issues

One -count=1 ./... run out of ten also failed, on TestRaft_SnapshotRestore_PeerChange and TestRaft_FollowerRemovalNoElection — both already documented. So the package does produce failures at -count=1; HasExistingState just was not among them in my sample.

What I have not established

I have not identified the mechanism. The extra server suggests state surviving between iterations of the package, but I have not confirmed that, and it could be in the test harness rather than in raft.

Nor can I rule out that HasExistingState fails at -count=1 at some rate below what 47 runs would detect. The honest claim is only that I did not observe it there.

Disclosure

Found while testing flakestat, a flaky-test detector I wrote, against real Go projects. raft was chosen because it is actively maintained and timing-dependent by nature, not because I suspected anything specific.

I got the characterisation wrong twice before arriving at the table above - first assuming the test leaked state into itself, then that shuffle ordering explained it. Both were ruled out by the isolation runs. Happy to run further experiments if any would help.