rafthttp: peerStatus permanently latches active=false when a pipeline POST fails after stream recovery

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via the report form.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

In server/etcdserver/api/rafthttp, each peer shares a single *peerStatus instance across 6 concurrent goroutines (msgAppV2Writer, writer, msgAppV2Reader, msgAppReader, pipeline, and snapSender).

All 6 workers call status.deactivate(...) on any I/O error, which sets peerStatus.active = false and peerStatus.since = time.Time{}. However, streamWriter and streamReader only call status.activate() once when a streaming HTTP connection is initially established (<-cw.connc in streamWriter.run() and cr.dial() in streamReader.run()). Steady-state streamWriter flushes and streamReader message/heartbeat reads never call status.activate().

As a result, peerStatus.active = false is an absorbing state whenever status.deactivate() runs after the streaming connections have already recovered:

1. At t = 0.0s, a transient TCP disruption (or peer restart) closes the leader's streamWriter (working = false). Before the stream re-attaches, peer.pick() routes 1–2 Raft messages (MsgHeartbeat / MsgApp) to p.pipeline.msgc. pipeline.handle() dials a new TCP connection to the peer, which stalls for peerDialTimeout (e.g. 2s) if that SYN or socket hits transient packet loss.
2. At t = +0.1s (DialRetryFrequency = 100ms), the peer's streamReader re-dials the leader, streamWriter attaches (working = true, so no further messages are sent to pipeline), and calls cw.status.activate() ("peer became active").
3. At t = +2.0s, the stale in-flight pipeline.post() times out and calls p.status.deactivate() ("peer became inactive (message send to peer failed)").

Because all subsequent Raft traffic flows exclusively over the already-established streamWriter and streamReader (which never call status.activate() again) and pipeline is never used while streamWriter.working == true, peerStatus.active remains false (activeSince() == time.Time{}) indefinitely—even though Raft replication and heartbeats are 100% healthy.

Impact:

• etcd_network_active_peers permanently under-reports active peers for that link.
• mayRemoveMember and mayAddMember (server/etcdserver/util.go) check tr.ActiveSince(id) and permanently reject self-removal (removing the member connected to the client endpoint) or member addition with etcdserver: unhealthy cluster in a 3-node cluster whenever 1 peer's peerStatus is latched active = false.

This is the underlying state-machine root cause of #14523 (which was auto-closed by the stale bot).

What did you expect to happen?

1. An in-flight pipeline.post() or snapshotSender.send() that started before a newer peerStatus activation (s.since.After(start)) should not clobber peerStatus.active or reset peerStatus.since when it later fails.
2. If peerStatus is deactivated while streamWriter or streamReader remains established, subsequent successful steady-state streamWriter flushes or unpaused streamReader message/heartbeat reads should transition peerStatus back to active = true.

How can we reproduce it (as minimally and precisely as possible)?

The script below deterministically reproduces the latch on a single Linux host with a 3-node local cluster (m1, m2, m3 on 127.0.0.11..13):

#!/usr/bin/env bash
# Copyright 2026 Google LLC.
# SPDX-License-Identifier: Apache-2.0
#
# Generic reproduction of the rafthttp peerStatus active=false latch on a single Linux host.
# Requirements: etcd, etcdctl, iproute2 (ss with CONFIG_INET_DIAG_DESTROY), iptables, curl.
set -euo pipefail

WORK_DIR=$(mktemp -d /tmp/etcd-latch-repro.XXXXXX)
IP1="127.0.0.11"
IP2="127.0.0.12"
IP3="127.0.0.13"
PEER_P=2380
CLIENT_P=2379
METRICS_P=2381

cleanup() {
  iptables -D OUTPUT -p tcp -s "$IP1" -d "$IP2" --dport "$PEER_P" -m conntrack --ctstate NEW -j DROP 2>/dev/null || true
  pkill -P $$ 2>/dev/null || true
  rm -rf "$WORK_DIR"
}
trap cleanup EXIT

CLUSTER="m1=http://${IP1}:${PEER_P},m2=http://${IP2}:${PEER_P},m3=http://${IP3}:${PEER_P}"

for i in 1 2 3; do
  IP="127.0.0.1${i}"
  etcd --name "m${i}" \
    --data-dir "${WORK_DIR}/m${i}.etcd" \
    --listen-peer-urls "http://${IP}:${PEER_P}" \
    --initial-advertise-peer-urls "http://${IP}:${PEER_P}" \
    --listen-client-urls "http://${IP}:${CLIENT_P}" \
    --advertise-client-urls "http://${IP}:${CLIENT_P}" \
    --listen-metrics-urls "http://${IP}:${METRICS_P}" \
    --initial-cluster "$CLUSTER" \
    --initial-cluster-token "repro-cluster" \
    --initial-cluster-state new \
    >"${WORK_DIR}/m${i}.log" 2>&1 &
done

sleep 4
M1_ID=$(curl -s "http://${IP1}:${METRICS_P}/metrics" | sed -n 's/.*Local="$[0-9a-f]*$".*/\1/p' | head -1)
etcdctl --endpoints="http://${IP1}:${CLIENT_P},http://${IP2}:${CLIENT_P},http://${IP3}:${CLIENT_P}" move-leader "$M1_ID"
sleep 6 # Wait >5s (ConnReadTimeout) so any bootstrap pipeline idle socket is closed by m2

# 1. Drop only NEW outbound TCP dials from m1 to m2:2380 (so pipeline dials hang for peerDialTimeout=2s,
#    while m1's ESTABLISHED streamReaders and m2->m1:2380 streamReader re-dials remain untouched).
iptables -I OUTPUT 1 -p tcp -s "$IP1" -d "$IP2" --dport "$PEER_P" -m conntrack --ctstate NEW -j DROP

# 2. Abort m1's inbound streamWriter sockets from m2 (m1:2380 <- m2).
#    - At t=+0.0s: m1's streamWriter closes and m1 routes 1-2 heartbeats to pipeline (whose NEW dial hangs for 2s).
#    - At t=+0.1s: m2's streamReader re-dials m1:2380; m1's streamWriter attaches and calls status.activate().
#    - At t=+2.0s: the in-flight pipeline dial times out and calls status.deactivate() LAST.
ss -t -K "src $IP1 sport = :$PEER_P dst $IP2"
sleep 3

# 3. Remove the iptables rule; network is 100% healthy, wait >5s (HealthInterval)
iptables -D OUTPUT -p tcp -s "$IP1" -d "$IP2" --dport "$PEER_P" -m conntrack --ctstate NEW -j DROP
sleep 6

echo "=== m1 log excerpt (shows 'became active' at +0.1s followed by pipeline timeout 'became inactive' at +2.0s) ==="
grep -E "became (in)?active|failed to write" "${WORK_DIR}/m1.log" | tail -n 6

echo "=== Metrics on m1 (active_peers=0 for m2 while has_leader=1 and Raft writes succeed) ==="
curl -s "http://${IP1}:${METRICS_P}/metrics" | grep -E '^etcd_network_active_peers|^etcd_server_has_leader'

echo "=== Attempting self MemberRemove via m1 (${IP1}:${CLIENT_P}) -> fails with 'etcdserver: unhealthy cluster' ==="
etcdctl --endpoints="http://${IP1}:${CLIENT_P}" member remove "$M1_ID"

Anything else we need to know?

No response

Etcd version (please run commands below)

Reproducible on main, release-3.6, release-3.5, and release-3.4.

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output