executor: IndexJoin OOM panic during inner result collection leaks the inner executor and worker
Bug Report
An OOM panic while collecting an IndexJoin inner result can skip closing the dynamically created inner executor. The query returns an error and RecordSet.Close() returns successfully, but an inner IndexLookup producer remains blocked on a channel send. Subsequent statements on the same session can reuse the original StatementContext while that old worker is still alive.
1. Minimal reproduce step (Required)
This is a self-contained mockstore reproduction with two synthetic tables. It uses three diagnostic failpoint callbacks: two observe inner IndexLookup start/close, and one lowers the result-list memory tracker limit immediately before innerResult.Add. The real chunk.List.Add -> Tracker.Consume -> PanicOnExceed path raises error 8175. It does not directly inject a panic or modify mutex/StatementContext state.
- Check out
2313f5f31336518c70c0061068129e9575b869e9in a disposable worktree. - Save the patch below as
repro.patchand rungit apply repro.patch. - Run:
./tools/check/failpoint-go-test.sh pkg/executor/test/distsqltest \
-run '^TestIndexJoinOOMInnerOOMLifecycle$' -count=3 -v -timeout=120sThe test should fail at inner executor survived query error and result-set Close. For the control, disable the quota injection:
IndexJoinOOM_CONTROL=normal ./tools/check/failpoint-go-test.sh \
pkg/executor/test/distsqltest \
-run '^TestIndexJoinOOMInnerOOMLifecycle$' -count=1 -v -timeout=120sThe control passes. The test explicitly closes retained executors during deferred cleanup so the reproduction itself does not leave workers behind after the test exits.
The SQL shape is an INL_JOIN whose inner table uses a non-covering secondary index. The fixture has 3 outer rows and 4096 inner rows. The full setup is included in the patch. This is an instrumented reproduction; a SQL-only reproduction using ordinary session/global limits has not yet been established.
diff --git a/pkg/executor/distsql.go b/pkg/executor/distsql.go
index 4c04323332..4ae58b65ce 100644
--- a/pkg/executor/distsql.go
+++ b/pkg/executor/distsql.go
@@ -29,6 +29,7 @@ import (
"unsafe"
"github.com/pingcap/errors"
+ "github.com/pingcap/failpoint"
"github.com/pingcap/tidb/pkg/distsql"
distsqlctx "github.com/pingcap/tidb/pkg/distsql/context"
"github.com/pingcap/tidb/pkg/executor/internal/builder"
@@ -734,6 +735,7 @@ func (e *IndexLookUpExecutor) startWorkers(ctx context.Context, initBatchSize in
return err
}
e.workerStarted = true
+ failpoint.InjectCall("indexJoinOOMLookupStarted", e)
return nil
}
@@ -1213,6 +1215,7 @@ func (e *IndexLookUpExecutor) Close() error {
e.finished = nil
e.workerStarted = false
e.resultCurr = nil
+ failpoint.InjectCall("indexJoinOOMLookupClosed", e)
return nil
}
diff --git a/pkg/executor/join/index_lookup_join.go b/pkg/executor/join/index_lookup_join.go
index e77bbc0e28..d0a8cba931 100644
--- a/pkg/executor/join/index_lookup_join.go
+++ b/pkg/executor/join/index_lookup_join.go
@@ -779,6 +779,7 @@ func (iw *innerWorker) fetchInnerResults(ctx context.Context, task *lookUpJoinTa
needClose = true
break
}
+ failpoint.InjectCall("indexJoinOOMBeforeInnerAdd", task.innerResult)
task.innerResult.Add(executorChk)
// If maxFetchSize is set, we need to break the loop when the fetched rows reach the max.
if iw.maxFetchSize > 0 && task.innerResult.Len() >= iw.maxFetchSize {
diff --git a/pkg/executor/test/distsqltest/BUILD.bazel b/pkg/executor/test/distsqltest/BUILD.bazel
index 2cf87858bd..e57bb6edf6 100644
--- a/pkg/executor/test/distsqltest/BUILD.bazel
+++ b/pkg/executor/test/distsqltest/BUILD.bazel
@@ -6,17 +6,23 @@ go_test(
srcs = [
"distsql_test.go",
"main_test.go",
+ "indexJoinOOM_repro_test.go",
],
flaky = True,
race = "on",
- shard_count = 2,
+ shard_count = 3,
deps = [
"//pkg/config",
"//pkg/config/kerneltype",
+ "//pkg/executor",
"//pkg/kv",
"//pkg/meta/autoid",
"//pkg/sessionctx/vardef",
"//pkg/testkit",
+ "//pkg/util/chunk",
+ "//pkg/util/memory",
+ "//pkg/util/sqlkiller",
+ "@com_github_pingcap_failpoint//:failpoint",
"@com_github_stretchr_testify//require",
"@com_github_tikv_client_go_v2//tikv",
"@org_uber_go_goleak//:goleak",
diff --git a/pkg/executor/test/distsqltest/indexJoinOOM_repro_test.go b/pkg/executor/test/distsqltest/indexJoinOOM_repro_test.go
new file mode 100644
index 0000000000..889dc987cd
--- /dev/null
+++ b/pkg/executor/test/distsqltest/indexJoinOOM_repro_test.go
@@ -0,0 +1,107 @@
+package distsql_test
+
+import (
+ "bytes"
+ "context"
+ "fmt"
+ "os"
+ "runtime/pprof"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/pingcap/failpoint"
+ "github.com/pingcap/tidb/pkg/executor"
+ "github.com/pingcap/tidb/pkg/testkit"
+ "github.com/pingcap/tidb/pkg/util/chunk"
+ "github.com/pingcap/tidb/pkg/util/memory"
+ "github.com/pingcap/tidb/pkg/util/sqlkiller"
+ "github.com/stretchr/testify/require"
+)
+
+// Diagnostic reproduction only: lower the real result-list tracker quota at a
+// precise allocation boundary. No mutex or StatementContext fields are modified.
+func TestIndexJoinOOMInnerOOMLifecycle(t *testing.T) {
+ store := testkit.CreateMockStore(t)
+ tk := testkit.NewTestKit(t, store)
+ tk.MustExec("use test")
+ tk.MustExec("create table a(id int primary key, customerid int, messageid int)")
+ tk.MustExec("create table m(id int primary key nonclustered, customerid int, body varchar(256), key ic(customerid))")
+ tk.MustExec("insert into a values(1,1,10),(2,1,20),(3,1,30)")
+ values := make([]string, 4096)
+ for i := range values {
+ values[i] = fmt.Sprintf("(%d,1,'%s')", i+1, strings.Repeat("x", 128))
+ }
+ tk.MustExec("insert into m values " + strings.Join(values, ","))
+ tk.MustExec("set tidb_index_lookup_size=32")
+ tk.MustExec("set tidb_index_lookup_concurrency=4")
+ tk.MustExec("set tidb_index_lookup_join_concurrency=1")
+ var mu sync.Mutex
+ active := map[*executor.IndexLookUpExecutor]bool{}
+ started := "github.com/pingcap/tidb/pkg/executor/indexJoinOOMLookupStarted"
+ closed := "github.com/pingcap/tidb/pkg/executor/indexJoinOOMLookupClosed"
+ inject := "github.com/pingcap/tidb/pkg/executor/join/indexJoinOOMBeforeInnerAdd"
+ require.NoError(t, failpoint.EnableCall(started, func(e *executor.IndexLookUpExecutor) { mu.Lock(); active[e] = true; mu.Unlock() }))
+ require.NoError(t, failpoint.EnableCall(closed, func(e *executor.IndexLookUpExecutor) { mu.Lock(); delete(active, e); mu.Unlock() }))
+ defer func() {
+ _ = failpoint.Disable(inject)
+ mu.Lock()
+ es := make([]*executor.IndexLookUpExecutor, 0, len(active))
+ for e := range active {
+ es = append(es, e)
+ }
+ mu.Unlock()
+ for _, e := range es {
+ _ = e.Close()
+ }
+ _ = failpoint.Disable(started)
+ _ = failpoint.Disable(closed)
+ }()
+ var once sync.Once
+ require.NoError(t, failpoint.EnableCall(inject, func(l *chunk.List) {
+ if os.Getenv("IndexJoinOOM_CONTROL") == "normal" {
+ return
+ }
+ once.Do(func() {
+ // Let prefetch fill resultCh while the consumer is at the allocation boundary.
+ time.Sleep(100 * time.Millisecond)
+ tr := l.GetMemTracker()
+ tr.SetActionOnExceed(&memory.PanicOnExceed{Killer: &sqlkiller.SQLKiller{}})
+ tr.SetBytesLimit(1)
+ })
+ }))
+ ctx := context.Background()
+ rs, err := tk.Exec("select /*+ INL_JOIN(m) */ a.id,m.body from a join m use index(ic) on a.customerid=m.customerid and a.messageid=m.id")
+ require.NoError(t, err)
+ sc := tk.Session().GetSessionVars().StmtCtx
+ chk := rs.NewChunk(nil)
+ for {
+ err = rs.Next(ctx, chk)
+ if err != nil || chk.NumRows() == 0 {
+ break
+ }
+ chk.Reset()
+ }
+ t.Logf("Next error: %v", err)
+ if os.Getenv("IndexJoinOOM_CONTROL") == "normal" {
+ require.NoError(t, err)
+ } else {
+ require.Error(t, err)
+ }
+ require.NoError(t, rs.Close())
+ _ = failpoint.Disable(inject)
+ mu.Lock()
+ leaked := len(active)
+ mu.Unlock()
+ t.Logf("inner IndexLookup executors not closed after statement Close: %d", leaked)
+ var stacks bytes.Buffer
+ require.NoError(t, pprof.Lookup("goroutine").WriteTo(&stacks, 2))
+ require.NoError(t, os.WriteFile(t.TempDir()+"/goroutines.txt", stacks.Bytes(), 0600))
+ t.Logf("IndexLookup producer stack still present: %v", strings.Contains(stacks.String(), "(*indexWorker).buildAndDispatchLookupTasks"))
+ tk.MustQuery("select 1").Check(testkit.Rows("1"))
+ tk.MustQuery("select 2").Check(testkit.Rows("2"))
+ reused := sc == tk.Session().GetSessionVars().StmtCtx
+ t.Logf("original StatementContext reused after two subsequent statements: %v", reused)
+ require.Zero(t, leaked, "inner executor survived query error and result-set Close")
+}2. What did you expect to see? (Required)
Once the query returns an OOM error and its result set is closed, all dynamically created inner executors should have been closed and their workers joined. No worker from the old statement should survive into reuse of its statement context.
3. What did you see instead (Required)
The observable bug is that the query has returned an OOM error and RecordSet.Close() has returned successfully, but a worker belonging to the old statement is still alive.
flowchart TB
A["Execute IndexJoin"] --> B["OOM while collecting inner results"]
B --> C["Query returns error 8175"]
C --> D["RecordSet.Close returns successfully"]
D --> E["Expected: inner executor closed<br/>Background workers exited"]
D --> F["Observed: 1 inner executor remains unclosed<br/>Old worker blocked on channel send"]
F --> G["Same session executes SELECT 1 and SELECT 2"]
G --> H["Original StatementContext is reused<br/>Old worker has not been cleaned up"]
style E fill:#dcfce7,stroke:#16a34a
style F fill:#fee2e2,stroke:#dc2626,stroke-width:3px
style H fill:#fee2e2,stroke:#dc2626,stroke-width:3px| State after the result set is closed | Normal control | OOM reproduction |
|---|---|---|
| Unclosed inner executors | 0 | 1 |
| Old IndexLookup producer stack still present | No | Yes |
| Original StatementContext reused by subsequent statements | Yes, after cleanup | Yes, while the old worker remains uncleaned |
Captured worker stack (local reproduction):
goroutine 10516 [chan send]:
(*indexWorker).buildAndDispatchLookupTasks
distsql.go:1625The diagram summarizes the observed lifecycle leak. It does not establish that StatementContext reuse caused a permanent mutex wait or a deadlock.
Repeated 3/3 times on the pinned source, after an initial successful reproduction:
Next error: [executor:8175]Your query has been cancelled due to exceeding the allowed memory limit ...
inner IndexLookup executors not closed after statement Close: 1
IndexLookup producer stack still present: true
original StatementContext reused after two subsequent statements: true
Error: Should be zero, but was 1
Messages: inner executor survived query error and result-set CloseThe goroutine dump shows the old indexWorker.buildAndDispatchLookupTasks blocked in a channel send. With quota injection disabled, the unclosed-executor count is 0, the producer stack is absent, and the test passes. The published patch only renames the diagnostic identifiers and changes the dump destination to a temporary directory relative to the executed reproduction; the execution logic is unchanged.
The relevant cleanup path remains in the master snapshot 12b639a1161cd5a60126a47277f5ad14c320fd4a:
fetchInnerResults:needClosestarts false. Ordinary error/cancellation/EOF set it true, but a panic frominnerResult.Addskips those assignments and the deferred inner close.innerWorker.run: recovers the panic and reports the error, then decrements the worker WaitGroup without closing the task's inner executor.IndexLookUpJoin.Close: waits for join workers and closes static children; the dynamically created inner executor is not covered by that static child tree.
Scope: this reproduces an executor/worker lifecycle leak and subsequent StatementContext reuse. It does not establish a permanent MergeCopExecDetails mutex wait, a lock cycle, or a complete production incident root cause.
Related: #51693 addressed a panic during inner executor construction. This report exercises a panic during result collection after the inner executor has been built and started.
4. What is your TiDB version? (Required)
- Executed reproduction: source commit
2313f5f31336518c70c0061068129e9575b869e9(reported deployment mapping: Premium v26.3.16). - Local environment: Go 1.25.12, darwin/arm64, testkit mockstore, build tags
intest,deadlock; this is not a deployed Premium/nextgen E2E test. - Master inspected:
12b639a1161cd5a60126a47277f5ad14c320fd4a(2026-09-29).fetchInnerResultsandinnerWorker.runare identical to the reproduced version. Master execution has not been verified in this report. - This is a source-level test, so no server
SELECT tidb_version()output is available.
Source: pingcap/tidb