coprocessor: full-sampling ANALYZE can time out in the read pool without execution
Bug Report
What version of TiKV are you using?
Both failures use the same TiKV build:
Release Version: 9.0.0-beta.2
Git Commit Hash: 3c8e5f454ff54d2e4765be3e079ec9481b0eb690
UTC Build Time: 2026-09-08 08:02:04Affected runs:
- TCMS plan 8205844, testbed
e2e-oltp-titan-tpcc-tps-8205844-1-557 - TCMS plan 8231190, testbed
e2e-oltp-tpcc-large-func-tps-8231190-1-411
What operating system and CPU are you using?
Kubernetes-based TCMS testbeds. OS, CPU model, and architecture were not captured in the case logs.
Steps to reproduce
Prepare a large TPCC dataset containing
tpcc20000.customer(about 600 million rows).Enable automatic Analyze tasks.
While automatic full-sampling Analyze tasks are running on large tables, execute:
ANALYZE TABLE tpcc20000.customer;Observe TiKV coprocessor logs and the result of the Analyze statement.
This was observed in two independent testbeds. A minimal deterministic reproduction has not yet been isolated.
What did you expect?
ANALYZE TABLE may take time under concurrent load, but full-sampling coprocessor requests should be scheduled and make progress. It should not fail with a TiKV timeout before any request processing occurs.
What did happened?
The manual Analyze failed in both runs with TiDB error [tikv:9002] TiKV server timeout:
2026-09-10 06:51:12 +08:00 ANALYZE TABLE tpcc20000.customer
2026-09-10 07:21:52 +08:00 analyze table `tpcc20000`.`customer` has failed
error="[tikv:9002]TiKV server timeout"2026-09-09 18:05:36 +08:00 ANALYZE TABLE tpcc20000.customer
2026-09-09 18:37:50 +08:00 analyze table `tpcc20000`.`customer` has failed
error="[tikv:9002]TiKV server timeout"The corresponding TiKV logs show that analyze_full_sampling requests expired while waiting for scheduling, without any processing time:
[tracker.rs:538] ["query deadline exceeded"]
[tag=analyze_full_sampling]
[total_process_time=0ns]
[wait_time.schedule=60.000835245s]
[total_lifetime=60.000848531s]The same signature was also observed for concurrent automatic Analyze tasks. There were no TiKV panic, OOM, or restart symptoms in either testbed.
For comparison, the same Analyze scenario succeeded on TCMS plan 8230370 with an earlier TiKV build:
Release Version: 9.0.0-beta.2
Git Commit Hash: 78aedc1c81ef3f7d8bacc6e9d09f56460f134937
UTC Build Time: 2026-08-31 06:40:13In the successful run, full-sampling requests experienced queue waits but kept making progress; the observed maximum wait was about 54 seconds and no query deadline exceeded was reported.
The first suspected TiKV functional change between the successful and failing builds is:
- #19854
*: batching analyze requests
This is only a hypothesis, not a confirmed root cause. The failure may be a regression or an interaction between batched/serial full-sampling Analyze handling and concurrent automatic Analyze workload. #19823 introduced a dedicated semaphore for full-sampling Analyze earlier; the expected isolation may be insufficient or bypassed under this workload.
Suggestions:
- Investigate scheduling/admission of batched and serial full-sampling Analyze tasks when manual and automatic Analyze run concurrently.
- Verify that requests do not spend their entire deadline in the read-pool scheduling queue with
total_process_time=0ns. - Add a regression test that runs concurrent automatic and manual full-sampling Analyze on large tables and verifies that queue starvation does not cause pre-execution timeout.
Source: tikv/tikv