TestRaft_HasExistingState fails intermittently under go test -count>1
TestRaft_HasExistingState fails intermittently when the package is run with
-count>1. I could not reproduce it at -count=1 in 47 attempts across three
scopes, so it looks specific to in-process repetition rather than something you
would hit in normal CI.
Raising it because -count=N with -shuffle is the usual way to hunt genuine
flakes, and this adds noise to that on the raft package. As far as I can tell it
is not in #372's list and has no existing issue, PR or commit.
Failure
--- FAIL: TestRaft_HasExistingState (5.20s)
testing.go:705: peer mismatch:
{Servers:[{Voter server-b3bfde62 ...} {Voter server-c21909b3 ...}]}
{Servers:[{Voter server-b3bfde62 ...} {Voter server-c21909b3 ...} {Voter server-1a02a2fe ...}]}Two servers expected, three observed — a node from elsewhere in the run appears in the configuration.
What I measured
At b853494d, go1.25.4, darwin/arm64. Every execution a fresh process.
| scope | flags | runs | HasExistingState failed |
|---|---|---|---|
| this test alone | -count=1 -shuffle |
25 | 0 |
| this test alone | -count=3 -shuffle |
25 | 0 |
| root package | -count=1 (no shuffle) |
12 | 0 |
| root package | -count=1 -shuffle |
12 | 0 |
./... |
-count=1 |
10 | 0 |
./... |
-count=3 -shuffle |
31 | 2 |
It needs the full package and -count>1. Seeds for the two failures:
758137489003, 837114709011.
Reproduction
go test -count=3 -shuffle=758137489003 ./...Roughly 1 run in 15, so expect to repeat it.
Other tests that failed in the same runs
All already known — listed only so the picture is complete, and because the rates may be useful:
| test | -count=3 runs |
note |
|---|---|---|
TestRaft_FollowerRemovalNoElection |
4/31 | #669 open, unmerged |
TestRaft_SendSnapshotFollower |
2/31 | #311, and in #372 |
TestRaft_ProtocolVersion_Upgrade_2_3 |
2/31 | in #372 |
TestRaft_RecoverCluster |
2/31 | in #372 |
TestRaft_ProtocolVersion_Upgrade_1_2 |
1/31 | in #372 |
TestNetworkTransport_AppendEntriesPipeline_CloseStreams |
1/31 | has prior issues |
One -count=1 ./... run out of ten also failed, on
TestRaft_SnapshotRestore_PeerChange and TestRaft_FollowerRemovalNoElection —
both already documented. So the package does produce failures at -count=1;
HasExistingState just was not among them in my sample.
What I have not established
I have not identified the mechanism. The extra server suggests state surviving between iterations of the package, but I have not confirmed that, and it could be in the test harness rather than in raft.
Nor can I rule out that HasExistingState fails at -count=1 at some rate
below what 47 runs would detect. The honest claim is only that I did not observe
it there.
Disclosure
Found while testing flakestat, a flaky-test detector I wrote, against real Go projects. raft was chosen because it is actively maintained and timing-dependent by nature, not because I suspected anything specific.
I got the characterisation wrong twice before arriving at the table above - first assuming the test leaked state into itself, then that shuffle ordering explained it. Both were ruled out by the isolation runs. Happy to run further experiments if any would help.
Source: hashicorp/raft