storage: TiKV scheduler freezes after memory quota fills up, keeps returning "scheduler is busy" (v8.5.6)

Bug Report

What version of TiKV are you using?

v8.5.6-20260508 (tidb-operator on EKS, raft-kv engine)

What operating system and CPU are you using?

Linux (EKS). storage.scheduler-worker-pool-size = 4, storage.memory-quota = 256MiB.

Steps to reproduce

Production incident; no minimal repro yet. Triggered under pessimistic-transaction writes with hot-key/region lock contention.

What did you expect?

After the scheduler memory quota fills up, it should drain as commands complete and recover on its own.

What did happen?

On a single node (tikv-7; the other 71 TiKV nodes were fine):

• tikv_scheduler_memory_quota_size{type="in_use"} climbed from ~tens of KB to 268,435,288 (=capacity 268,435,456, i.e. 100%) within ~4 minutes (14:39–14:43), then stayed pegged for 2+ hours without draining.
• tikv_scheduler_contex_total (pending commands) froze at 233,485; tikv_scheduler_running_commands froze at 15,052; tikv_scheduler_write_flow dropped to 0.
• All tikv_scheduler_stage_total stages stopped incrementing after 14:43; scheduler worker CPU dropped to 0.
• New writes were continuously rejected with ServerIsBusy (reason="scheduler is busy"); rate(tikv_scheduler_too_busy_total[5m]) stayed at ~4,400/s for acquire_pessimistic_lock and ~7,000/s for pessimistic_rollback.
• Point reads (kv_get/kv_batch_get) still returned normally (~0.2ms latency); raft/raftstore kept working — only the transaction write path (scheduler) was frozen.

Ruled out: write-QPS burst (prewrite/commit QPS was steady ~1,300/s before dropping to 0), flow-control/L0 throttling, RocksDB write stall, and raftstore apply stall.

Recovery required restarting tikv-7.