raft: Joining node can self-remove permanently during initial log replay

Describe the bug

The removed checker introduced in #28726 and #29052 causes node to remove itself permanently during initial log replay on slow or CPU-constrained systems.

To Reproduce

Assume a 3-node cluster: vault-0, vault-1 and vault-2. The following sequence of events occurs during initial cluster setup:

1. vault-0 becomes the leader.

2. vault-1 joins the cluster.

3. vault-2 joins the cluster.

   Log records from the vault-0 (leader) perspective:

   复制

   $ raft-inspector log  LogConfiguration --data-dir data/vault-0
   ─── Index 1 (raft/raft.db logs/1) ───
     Index:      1
     Term:       1
     Type:       LogConfiguration
     AppendedAt: 0001-01-01 00:00:00 +0000 UTC
     Servers:
       - vault-0 (vault-0:8201) voter
   
   ─── Index 38 (raft/raft.db logs/38) ───
     Index:      38
     Term:       3
     Type:       LogConfiguration
     AppendedAt: 2026-08-26 08:03:46.82551766 +0000 UTC  (+0s)
     Servers:
       - vault-0 (vault-0:8201) voter
       - vault-1 (vault-1:8201) nonvoter
   
   ─── Index 289 (raft/raft.db logs/289) ───
     Index:      289
     Term:       3
     Type:       LogConfiguration
     AppendedAt: 2026-08-26 08:03:58.502445435 +0000 UTC  (+11.677s)
     Servers:
       - vault-0 (vault-0:8201) voter
       - vault-1 (vault-1:8201) voter
   
   ─── Index 290 (raft/raft.db logs/290) ───
     Index:      290
     Term:       3
     Type:       LogConfiguration
     AppendedAt: 2026-08-26 08:04:17.093120765 +0000 UTC  (+30.268s)
     Servers:
       - vault-0 (vault-0:8201) voter
       - vault-1 (vault-1:8201) voter
       - vault-2 (vault-2:8201) nonvoter

4. vault-2 receives the member list from vault-0 when joining.

5. vault-2 calls raft.BootsrapCluster() at initial unseal which stores a fabricated entry in the vault-2 log (link, link).

   From raft.db on vault-2:

   复制

   $ raft-inspector log  1 --data-dir data/vault-2 
     Index:      1
     Term:       1
     Type:       LogConfiguration
     AppendedAt: 0001-01-01 00:00:00 +0000 UTC
     Servers:
       - vault-0 (vault-0:8201) voter
       - vault-1 (vault-1:8201) voter
       - vault-2 (vault-2:8201) nonvoter

   At this point vault-2 satisfies the first criteria used for self-removal: vault-2 has been present in the configuration (link).

6. Log entries replicate from vault-0 to vault-2. It processes index 38, where vault-1 joined.

   复制

   $ raft-inspector log 38 --data-dir data/vault-2
     Index:      38
     Term:       3
     Type:       LogConfiguration
     AppendedAt: 2026-08-26 08:03:46.82551766 +0000 UTC  (+0s)
     Servers:
       - vault-0 (vault-0:8201) voter
       - vault-1 (vault-1:8201) nonvoter

   At this point, vault-2satisfies the second crieria: it is removed in the configuration (link).

7. The periodic "removed checker" runs and marks the node for removal because both criteria are met.

   The CPU running vault-2 is busy and/or throttled, log replay is slow, and index 290 (where vault-2 joined) has not been reached yet. As a result, the configuration vault-2 has still only lists vault-0 and vault-1.

8. Result is permanent shutdown

   复制

   core: shutting down core: error="node has been removed from the HA cluster"

   vault-2 has persisted removed=1 on raft.db conf bucket, and it will refuse to work after that (link)

   复制

   $ raft-inspector status --data-dir data/vault-2 | grep "Removed Self"
     Removed Self:       1

Expected behavior

The cluster should become healthy with all nodes successfully joined.

Environment:

• Vault Server Version: 1.19 and later

Additional context I will come up with repro after a bit.

The tool raft-inspector is my own, you can find it here.