#71107·tidb

ingestor: failed Pebble opens leak disk health checking goroutines

Author: wjhuang2016Created Sep 15, 2026Updated Sep 17, 2026
Labelstype/bugseverity/majorcomponent/lightningaffects-8.1affects-8.5affects-26.3affects-25.10

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

TiDB's local ingest backend opens Pebble databases with a nil pebble.Options.FS, so Pebble installs its default disk-health-checking filesystem. If pebble.Open fails after a filesystem metadata operation has started the health-check ticker, the returned DB is nil and TiDB has no handle through which to close the filesystem.

Using TiDB's pinned Pebble dependency (v1.1.4-0.20250120151818-5dd133a1e6fb):

  1. Repeatedly call pebble.Open 100 times for each of these expected failure paths:
    • the parent path is a regular file;
    • another DB holds the directory lock;
    • the database exists and ErrorIfExists is enabled.
  2. Count goroutines whose top function is github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1.
  3. Wait longer than the ticker interval and run GC.

Each failed open leaves one disk-health-checking goroutine behind. Across the three cases, 300 failed opens leave 300 goroutines. They remain after waiting and GC.

TiDB has two production pebble.Open call sites in pkg/ingestor/ingestctrl/engine_mgr.go, used by local ingest engines and the duplicate-detection DB. Repeated import or DDL ingest failures can therefore accumulate these goroutines in a long-running TiDB process.

This is the same ownership problem fixed upstream by cockroachdb/pebble#5787, but TiDB's pinned Pebble revision does not contain that change.

2. What did you expect to see? (Required)

When a local ingest Pebble DB fails to open, all filesystem resources created for that attempt should be released. Repeated failures should not increase the number of disk-health-checking goroutines. Successful DB opens must retain disk-health monitoring until DB.Close.

3. What did you see instead (Required)

Every failed open can leave one diskHealthCheckingFS.startTickerLocked.func1 goroutine. The goroutine keeps the filesystem wrapper reachable, so GC cannot reclaim it. At very large counts, goroutine profile collection can also cause long non-GC process pauses; pingcap/tidb#62080 reduces repeated profile collection but does not fix this leak.

4. What is your TiDB version? (Required)

Reproduced with the Pebble revision pinned by TiDB master and release-8.5:

github.com/cockroachdb/pebble v1.1.4-0.20250120151818-5dd133a1e6fb

The issue was also observed on TiDB v8.5.6.

Related upstream change: https://github.com/cockroachdb/pebble/pull/5787