[Enhancement] Write flow control can turn the throttled store into a read hotspot for non-leader reads
Scope: only workloads that actually issue non-leader reads (follower read / stale read /
ReplicaReadMixed / ReplicaReadPreferLeader). With leader-only reads — the default — the store keeps
receiving the same read traffic and this problem does not occur; see root cause (2).
Phenomenon
When a store's write flow control kicks in (pending_compaction_bytes above the soft limit), that store
ends up being loaded with read hotspots even though its reads were never degraded:
- Non-leader reads start avoiding the store, so its measured read load collapses — even though the store serves reads normally.
- PD's hot-read scheduler ranks stores by measured read load, so the store now looks like the coldest
one and becomes a destination:
transfer-hot-read-leader/move-hot-read-leader/move-hot-read-peermove read hotspots onto it, andmove-hot-read-leaderalso creates a new replica there. - When compaction catches up and throttling stops, the store is hit by all of that at once and can be saturated by read traffic.
Net effect: a write-only degradation produces a read hotspot on the same store — but only for workloads whose read traffic is eligible to be diverted away from the leader.
Root cause
TiKV — the rejection is write-only and carries no wait estimate. The flow controller's
should_drop()(src/storage/txn/flow_controller/singleton_flow_controller.rs) rejects commands inrun_cmdonly whenneed_flow_control()is true, i.e.!readonly && priority != High— reads are never rejected this way. The error iserrorpb::ServerIsBusy { reason: "scheduler is busy" }withestimated_wait_ms == 0(src/storage/errors.rs; the throttle-wait path returns"deadline is exceeded", also 0).client-go — a write rejection becomes a store-wide penalty that only non-leader reads can act on. In
onServerIsBusy(internal/locate/replica_selector.go), theestimated_wait_ms != 0branch distinguishes reads withisReadReq(req.Type), but the== 0branch — the one flow-control errors take — callsctx.Store.healthStatus.markAlreadySlow()regardless of request type. That pins the sharedStoreobject's client-side slow score to its maximum.That score only gates non-leader replica selection:
if !r.store.healthStatus.IsSlow() { score |= flagNotSlow }in the mixed strategy, used by follower / mixed / stale / prefer-leader reads;- under prefer-leader, non-leader replicas on a slow store are skipped outright.
Leader-forced reads are not affected.
isLeaderCandidatedoes not consultIsSlow, and the leader-read diversion path also requiresEstimatedWaitTimeorserverIsBusyFlag, neither of which the flow-control error sets. So with default leader reads the store keeps getting the same read traffic, no read-load collapse happens, and PD never sees it as cold.PD — the collapse is measured and acted on. Store read load is the time-median of the store heartbeat's
bytes_read(pkg/statistics/store.go), andbytes_readis the read flow the store actually served (components/raftstore/src/store/worker/pd.rs,handle_read_stats). A store whose non-leader read traffic was diverted therefore satisfies the hot-read destination test (maxLoad * DstToleranceRatio(1.05) < cluster expectation) by construction.
Source: tikv/tikv