#20076·tikv

[Enhancement] Write flow control can turn the throttled store into a read hotspot for non-leader reads

Author: mayjiang0203Created Sep 11, 2026Updated Sep 11, 2026
Labelstype/enhancementcontribution

Scope: only workloads that actually issue non-leader reads (follower read / stale read / ReplicaReadMixed / ReplicaReadPreferLeader). With leader-only reads — the default — the store keeps receiving the same read traffic and this problem does not occur; see root cause (2).

Phenomenon

When a store's write flow control kicks in (pending_compaction_bytes above the soft limit), that store ends up being loaded with read hotspots even though its reads were never degraded:

  1. Non-leader reads start avoiding the store, so its measured read load collapses — even though the store serves reads normally.
  2. PD's hot-read scheduler ranks stores by measured read load, so the store now looks like the coldest one and becomes a destination: transfer-hot-read-leader / move-hot-read-leader / move-hot-read-peer move read hotspots onto it, and move-hot-read-leader also creates a new replica there.
  3. When compaction catches up and throttling stops, the store is hit by all of that at once and can be saturated by read traffic.

Net effect: a write-only degradation produces a read hotspot on the same store — but only for workloads whose read traffic is eligible to be diverted away from the leader.

Root cause

  1. TiKV — the rejection is write-only and carries no wait estimate. The flow controller's should_drop() (src/storage/txn/flow_controller/singleton_flow_controller.rs) rejects commands in run_cmd only when need_flow_control() is true, i.e. !readonly && priority != High — reads are never rejected this way. The error is errorpb::ServerIsBusy { reason: "scheduler is busy" } with estimated_wait_ms == 0 (src/storage/errors.rs; the throttle-wait path returns "deadline is exceeded", also 0).

  2. client-go — a write rejection becomes a store-wide penalty that only non-leader reads can act on. In onServerIsBusy (internal/locate/replica_selector.go), the estimated_wait_ms != 0 branch distinguishes reads with isReadReq(req.Type), but the == 0 branch — the one flow-control errors take — calls ctx.Store.healthStatus.markAlreadySlow() regardless of request type. That pins the shared Store object's client-side slow score to its maximum.

    That score only gates non-leader replica selection:

    • if !r.store.healthStatus.IsSlow() { score |= flagNotSlow } in the mixed strategy, used by follower / mixed / stale / prefer-leader reads;
    • under prefer-leader, non-leader replicas on a slow store are skipped outright.

    Leader-forced reads are not affected. isLeaderCandidate does not consult IsSlow, and the leader-read diversion path also requires EstimatedWaitTime or serverIsBusyFlag, neither of which the flow-control error sets. So with default leader reads the store keeps getting the same read traffic, no read-load collapse happens, and PD never sees it as cold.

  3. PD — the collapse is measured and acted on. Store read load is the time-median of the store heartbeat's bytes_read (pkg/statistics/store.go), and bytes_read is the read flow the store actually served (components/raftstore/src/store/worker/pd.rs, handle_read_stats). A store whose non-leader read traffic was diverted therefore satisfies the hot-read destination test (maxLoad * DstToleranceRatio(1.05) < cluster expectation) by construction.