ingestor: failed Pebble opens leak disk health checking goroutines
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
TiDB's local ingest backend opens Pebble databases with a nil pebble.Options.FS, so Pebble installs its default disk-health-checking filesystem. If pebble.Open fails after a filesystem metadata operation has started the health-check ticker, the returned DB is nil and TiDB has no handle through which to close the filesystem.
Using TiDB's pinned Pebble dependency (v1.1.4-0.20250120151818-5dd133a1e6fb):
- Repeatedly call
pebble.Open100 times for each of these expected failure paths:- the parent path is a regular file;
- another DB holds the directory lock;
- the database exists and
ErrorIfExistsis enabled.
- Count goroutines whose top function is
github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1. - Wait longer than the ticker interval and run GC.
Each failed open leaves one disk-health-checking goroutine behind. Across the three cases, 300 failed opens leave 300 goroutines. They remain after waiting and GC.
TiDB has two production pebble.Open call sites in pkg/ingestor/ingestctrl/engine_mgr.go, used by local ingest engines and the duplicate-detection DB. Repeated import or DDL ingest failures can therefore accumulate these goroutines in a long-running TiDB process.
This is the same ownership problem fixed upstream by cockroachdb/pebble#5787, but TiDB's pinned Pebble revision does not contain that change.
2. What did you expect to see? (Required)
When a local ingest Pebble DB fails to open, all filesystem resources created for that attempt should be released. Repeated failures should not increase the number of disk-health-checking goroutines. Successful DB opens must retain disk-health monitoring until DB.Close.
3. What did you see instead (Required)
Every failed open can leave one diskHealthCheckingFS.startTickerLocked.func1 goroutine. The goroutine keeps the filesystem wrapper reachable, so GC cannot reclaim it. At very large counts, goroutine profile collection can also cause long non-GC process pauses; pingcap/tidb#62080 reduces repeated profile collection but does not fix this leak.
4. What is your TiDB version? (Required)
Reproduced with the Pebble revision pinned by TiDB master and release-8.5:
github.com/cockroachdb/pebble v1.1.4-0.20250120151818-5dd133a1e6fbThe issue was also observed on TiDB v8.5.6.
Related upstream change: https://github.com/cockroachdb/pebble/pull/5787
Source: pingcap/tidb