[import] pre-opened stat readers can leak on iterator initialization failure or early close
Bug Report
1. Minimal reproduce step (Required)
This issue was found by AI-assisted static analysis while reviewing #70942. It covers two ownership failures in the asynchronous stat-reader pre-open pipeline used by newMergePropBaseIter.
Initialization-error path
- Build a
MultipleFilesStatwith more than2 * limitstat files sonewMergePropBaseIterstarts asynchronous pre-opens and can fillpreOpenCh. - Make one of the first
limitsynchronousNewStatsReadercalls fail while asynchronous opens for later files succeed. newLimitSizeMergeIterreturns an error, andnewMergePropBaseIterreturns a half-built iterator together with that error.NewMergePropIterdiscards the iterator without closingcloseCh. The producer can remain blocked onpreOpenCh, and successfully pre-opened readers have no remaining owner.
Early-close path after the producer exits
- Build a
MultipleFilesStatwithlen(Filenames) <= 2 * limitand make every stat-reader open succeed. - Wait until the producer has put all asynchronous task channels into
preOpenChand exited, without consuming those tasks through the iterator. - Call
mergePropBaseIter.close(). close()waits for the finished producer and closes readers owned bym.iter, but it never drains the successful readers still buffered behindpreOpenCh.
Relevant review findings:
- https://github.com/pingcap/tidb/pull/70942#discussion_r3957158973
- https://github.com/pingcap/tidb/pull/70942#discussion_r3957158982
The relevant producer, constructor return, and close logic predates #70942 and was introduced by #49832.
2. What did you expect to see? (Required)
Reader ownership should be resolved on every constructor and shutdown path. If iterator construction fails or the iterator closes before consuming all pre-opened tasks, all producer goroutines should terminate and every successfully opened StatsReader should be closed exactly once.
3. What did you see instead (Required)
On an initialization error, the half-built iterator is discarded without signaling closeCh. Depending on the number of files, the producer either blocks indefinitely after filling preOpenCh or exits while successful reader results remain unconsumed. In both cases, pre-opened readers can leak.
On early close after the producer has already exited, the only preOpenCh drain is no longer reachable. mergePropBaseIter.close() does not drain the remaining tasks, so their readers stay open together with their storage connections and prefetch buffers.
4. What is your TiDB version? (Required)
Confirmed by source inspection on PR #70942's base commit 10f06594e7e60c2238a951d9bc151132b6f5f0c0. The ownership structure traces back to commit 8a79c0d56c6f7f9570ba6d4697ace1f26ce46b45 from December 2023.
Not dynamically reproduced against a released TiDB version or real S3/GCS service yet.
Suggested fix
- Make
newMergePropBaseIterself-cleaning whennewLimitSizeMergeIterreturns an error, and returnnil, errrather than an unusable half-built iterator. - Give
close()unconditional ownership of draining any task channels left inpreOpenChafter the producer stops, while ensuring a reader is closed exactly once. - Consider
ctx.Done()as an additional producer exit signal, but do not rely on cancellation alone to release readers that were already opened. - Add deterministic tests that track open-reader references for both an initialization failure with a full channel and an early close after successful pre-opens.
Parent issue
- #69798
Source: pingcap/tidb