etcdserver: v3.7 leader sends MsgApp to peers in reverse log order
Not security related. Not a support request. No existing report of this send-order bug.
What happened?
On v3.7.0, v3.7.1, and main, processMessages walks Ready messages last to first and appends the ones it keeps, so MsgApp leaves the leader in reverse log order. v3.6 returned the original slice.
A follower rejects a MsgApp that arrives before the index it depends on. The leader lowers Next and resends. Those rejects are not rafthttp send failures.
Introduced by f3cea0a978 (#21810), kept in 4715a5474c (#21778). raft still emits Ready.Messages in log order.
What did you expect to happen?
Survivors leave in Ready order. Keep the backward filter (removed members, older MsgAppResp, MsgSnap). Do not set To = 0 on the inputs; they are pointers into raft state.
Please backport to release-3.7. v3.6 is not affected.
How can we reproduce it (as minimally and precisely as possible)?
Call processMessages the same way TestProcessDuplicatedAppRespMessage does:
- MsgApp to peer 2, indexes 10, 11, 12
- MsgAppResp to peer 1, indexes 1 then 2
- MsgApp to removed peer 5
On main the kept messages come back reversed (MsgApp 12, 11, 10, then MsgAppResp 2). Expected: MsgApp 10, 11, 12, then MsgAppResp 2. Inputs are unchanged.
https://github.com/etcd-io/etcd/blob/main/server/etcdserver/raft.go#L357-L401
Anything else we need to know?
On a cluster under sustained writes, caught-up followers receive each entry about 2.5 to 3 times. A slower member's resends fill the 4096-slot rafthttp send buffer ("sending buffer is full"). The cluster stays up, but clients see high latency, failed proposals, follower lag, and snapshots. v3.6.13 on the same workload does not.
With 12 KB values on a 5-member cluster, v3.7.1 sustained about half the put rate of v3.6.13. Restoring send order matched v3.6.13.
Etcd version
main, v3.7.0, and v3.7.1. The order check above is the repro.
Etcd configuration
N/A
Etcd debug information
N/A
Relevant log output
N/A
Source: etcd-io/etcd