Metastore raft cluster does not self-heal after single-node disk write stall — requires manual pod restart

Describe the bug

In a 3-node metastore raft cluster, a transient write-latency spike on a single node's PVC triggered the timeoutLogStore write timeout (added in #4892/#4935), causing that node to step down as leader. Instead of the cluster electing a new leader and recovering within seconds, the affected node got stuck reporting node is not the leader / failing readiness probes for several minutes, and other components (ingesters, compaction workers) kept round-robining requests into the stuck node via metastore-client. The cluster only recovered once the affected pod was manually restarted — raft did not self-heal on its own despite 2 of 3 nodes remaining healthy throughout.

This looks like a real-world hit of the mechanism described in #5307 ("fix(metastore): fix follower livelock from abandoned log-store writes", currently draft/unmerged): the timeout wrapper aborts the wait on a slow StoreLogs call but can't cancel the underlying write, which keeps running in the background and can still land later, violating raft-wal's single-writer/monotonicity invariants and leaving the node in a half-broken "zombie" state.

To Reproduce

1. Start Pyroscope metastore (version/SHA: grafana/pyroscope:2.2.0), 3-node raft cluster, each backed by its own PVC.
2. Induce a write-latency spike on one node's PVC sufficient to exceed the metastore's LogStoreTimeout (default 10s) on a StoreLogs call — in our case this occurred organically from underlying storage contention, not a fault injection.
3. Observe the affected node log log store write timed out after 10s and step down as leader (raft: failed to commit logs).
4. Observe kubelet readiness probes fail on that pod, and other components see rpc error: code = Unavailable desc = node is not the leader while round-robining across all 3 metastore servers.

Expected behavior

The two unaffected nodes should elect a new leader and the cluster should return to a healthy state within a normal raft election window (single-digit seconds), even if the node that experienced the disk stall takes longer to fully recover. Downstream clients should converge on the new leader quickly rather than repeatedly retrying the stuck node.

Instead, in our incident:

• Metastore-0 and metastore-2 recovered readiness within seconds of the leadership change.
• Metastore-1 (the node that experienced the stall) continued failing readiness probes roughly every 10s for ~3.5 more minutes, and continued being selected by metastore-client's random leader-selection logic on other components' retries.
• The cluster only fully recovered after metastore-1's pod was manually restarted.

Environment

• Infrastructure: Kubernetes (RKE2, Hetzner-backed bare-metal workers)
• Deployment tool: Helm
• Metastore: 3 replicas, each with a dedicated PVC (StatefulSet pyroscope-metastore)

Additional Context

Timeline of the incident (all times UTC, 2026-07-28):

Time	Event
06:09:00–06:11:00	Average write latency on the affected node's PVC spikes from a ~2.3ms baseline to 183–225ms (2-min average) — consistent with an underlying multi-second write stall
06:09:59–06:10:02	Kubelet readiness probes fail (context deadline exceeded) on all three metastore pods
06:10:04	msg="log store write timed out after 10s" → raft: failed to commit logs → affected node steps down as leader
06:10:04–~06:13:28	The two unaffected nodes recover readiness within seconds; the affected node alone continues failing readiness probes every ~10s
~06:13:00	Affected pod restarted manually; cluster recovers immediately

Node-level health (DiskPressure, MemoryPressure, Ready conditions, node-exporter scrape continuity) stayed clean throughout — this was an isolated storage-layer event on one node's volume, not a node eviction/pressure event.

Representative log lines: level=error caller=raftnode msg="failed to commit logs" error="log store write timed out after 10s" level=warn caller=grpc_logging.go method=/raft_node.RaftNodeService/ReadIndex duration=10.000998033s err="rpc error: code = Unavailable desc = node is not the leader" level=error caller=methods.go component=metastore-client msg="metastore client error" err="rpc error: code = Unavailable desc = node is not the leader" server_id=pyroscope-metastore-1... level=info caller=methods.go component=metastore-client msg="changing metastore client leader" current=pyroscope-metastore-0... new=pyroscope-metastore-1...

Related:

• #5307 — draft fix for the abandoned-write livelock mechanism; this issue may be a good real-world trace to validate it against.
• pkg/metastore/client/methods.go's selectInstance() has no way to temporarily deprioritize a server that just errored — it always re-selects uniformly at random among all servers, which likely extended the practical outage window for downstream callers beyond what the raft layer alone caused.
• #5257 — same error strings but a different root cause (single-replica v2.0.3 regression), not a duplicate.