#20019·tikv

resource_control: foreground CPU throttling penalizes tenants that did not cause the overload

Author: mittalrishabhCreated Aug 23, 2026Updated Sep 7, 2026
Labelstype/bugseverity/moderate

Bug Report

What version of TiKV are you using?

master (8e964719d). Observed in production on a 8.5-based build carrying the same code path.

What operating system and CPU are you using?

Linux x86_64, 32 vCPU (AWS m6i.8xlarge).

Steps to reproduce

  1. Two resource groups on one TiKV node, both with an established historical baseline.
  2. Drive a short read burst from group A so its rate rises ~10x above its own baseline, and let group B rise ~2x above its own baseline at the same time.
  3. Let foreground CPU cross fg_cpu_throttle_threshold so adjust_group_throttling engages.

What did you expect?

The group responsible for the overload is throttled and deprioritized. The group that barely moved keeps running unthrottled — it did not cause the pressure and clamping it does not meaningfully relieve the node.

What did happen?

Both groups were throttled and deprioritized. adjust_group_throttling and deprioritize_over_quota_groups evaluate each group in isolation:

rust
let over_quota = hist > 0.0 && current > hist * burst_factor;

There is no comparison between groups and no estimate of how much CPU relief each one would provide. Any group above its own burst target is clamped, however small its share of the overload.

Impact seen on the group that did not cause the spike: p99 rose from ~11 ms to ~58 ms and stayed there for ~90 s while its CPU limit was in force, despite its total resource usage being a small fraction of the noisy group's. Because a resource group exists to isolate tenants from each other, penalizing an uninvolved tenant defeats the purpose of the feature.

Proposed fix

Make the decision relative: sum current_rate across groups, keep the baseline_burst_pct test as the eligibility gate, sort candidates by excess over their own baseline, and take from the top only until that excess covers the node's overshoot — always taking at least the top candidate so an overloaded tick never spares everyone. Both call sites then share one selection instead of each re-deriving over-quota per group.

PR: #20020