etcdserver: leader apply backlog fails to recover after a transient stall

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via the report form.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Past a certain backlog depth, apply throughput on the leader drops and does not come back, so the backlog keeps growing on its own. I only ever saw this on the leader, never on a follower.

I track the apply backlog with:

• etcd_server_proposals_committed_total - etcd_server_proposals_applied_total

Once the backlog reaches maxGapBetweenApplyAndCommitIndex = 5000, the member starts rejecting requests, the Kubernetes cluster in front of it becomes unstable, and leaving it alone does not make things better. And RSS grows along with the backlog and eventually leads to OOM. The only workaround I found was to reduce the incoming load and restart the leader.

I first saw this on a cluster running etcd v3.5.21 (a17edfd, kernel 5.15). What was puzzling is that nothing stood out on the physical host beforehand(no disk I/O problem). Then I experimented on etcd v3.6.6 (d2809cf, kernel 6.12) and managed to reproduce a very similar failure. Both are large Kubernetes clusters at a similar scale, with the test cluster built from hollow-nodes. Everything below comes from the test cluster.

What triggers it

• My current working hypothesis is that this is the effect of mmap memory reclaim caused by NUMA node pressure (or by a unknown disk problem), and I was able to reproduce it deliberately with a few tools. I saw mmap memory reclaim happen even though the host looked like it had plenty of free pages, and observed that this was NUMA node imbalance with one node short on free pages.
  • Even with no test load applied, I saw the inactive_file cache grow naturally, put pressure on a NUMA node, and get as far as etcd major page faults.
  • MongoDB and MySQL document similar caveats about running on NUMA hardware:
    • https://www.mongodb.com/docs/manual/administration/production-notes/#mongodb-and-numa-hardware
    • http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/
• The sequence is:
  1. One NUMA node comes under memory pressure, and the kernel reclaims file-backed pages belonging to the bbolt mmap.
  2. When etcd touches a reclaimed page it takes a major page fault, and since the page is no longer in the page cache it has to be read back from disk.
  3. MVCC compaction holds the kvstore (bbolt) batchTx lock.
  4. During this I observed per-batch db compaction pauses of up to ~173ms, and ~4.48min of db compaction total.
  5. For the apply requests that go through TxnWrite, apply index throughput drops, and when it does not recover on its own it amplifies itself.
  6. The problem I want to address is that last part: it does not recover on its own.

What did you expect to happen?

After a transient apply delay ends,

• apply throughput should catch back up with the incoming proposal rate,
• and the backlog should fall back to the small,
• steady gap it normally sits at, without operator intervention.
  • keep under maxGapBetweenApplyAndCommitIndex = 5000

How can we reproduce it (as minimally and precisely as possible)?

It does not happen deterministically, but the reproduction rate goes up with enough load and strong enough NUMA node pressure. I reproduced it several times with the steps below. The benchmark options I used are attached.

1. Keep a v3.6.6 cluster at roughly 897 committed/s and ~4.47k rpc ops/s on the leader using etcd benchmark:
   • txn-put: raises the committed/applied index and makes mvcc compaction contend for the batchTx lock,
   • linearizable read(range): grows the number of applyWait waiters,
   • lease keepalive: grows the number of entries in the timeList map.
2. Apply memory pressure to a single NUMA node. On a 128 GB host with two 64 GB nodes, allocate ~62 GB on one of them so that kswapd and direct reclaim evict etcd's mmap pages.
   • Simply letting the inactive_file cache grow naturally was also enough to push the node into reclaim and produce etcd major page faults.
3. Keep the memory pressure short.
4. From here the gap grows linearly and the apply rate (rate(etcd_server_proposals_applied_total[1m])) collapses. After that mvcc compaction cannot make progress, the revision stays in the past, and RSS keeps climbing.

To see state I could not otherwise observe, I added three debugging metrics locally:

The backlog starts growing at exactly the moment per-batch Trigger time starts to rise. After that you can also see the scheduled batches falling behind.

put committed & applied proposals

/usr/local/go/bin/benchmark \
-- endpoints=${EPS} \
--conns=40 \
--clients=200 \
txn-put \
--key-size=32 \
--key-space-size=20000 \
--val-size=10240 \
--rate=600 --total=1000000000

increase read to amplify waits

• read key whatever

/usr/local/go/bin/benchmark \
--endpoints=${EPS} \
--target-leader \
--conns=20 \
--clients=200 \
range benchmark-read-key \
--consistency=l \
--total=100000000

put lease-keepalive to increase entries

/usr/local/go/bin/benchmark \
--endpoints=${EPS} \
--conns=50 \
--clients=1000 \
lease-keepalive \
--total=1000000000

Anything else we need to know?

I'm looking at the growing backlog and the growing memory as two separate problems, and I'm currently testing patches for both. I'd like to discuss this together.

This is the first time I have looked into etcd this deeply, so please tell me if I have got something wrong, or if there is anything else you need.

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.6.6
Git SHA: d2809cf00
Go Version: go1.24.10
Go OS/Arch: linux/amd64

$ etcdctl version
"level":"info","ts":"2026-09-19T13:32:21.768216+0900","caller":"etcdmain/etcd.go:64","msg":"Running: ","args":["/usr/bin/etcd","version"]}
{"level":"warn","ts":"2026-09-19T13:32:21.768753+0900","caller":"etcdmain/etcd.go:66","msg":"failed to verify flags","error":"\"version\" is not a valid flag"}

kernel 6.12

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output