#53538 [Bug]: [3.0][stability] Search fails with "Future was cancelled" (cardinal & faiss IVF) during concurrent delete+compact churn

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: v3.0.1-nightly-5c761ffbf (3.0 branch, 2026-09-15 nightly), commit https://github.com/milvus-io/milvus/commit/5c761ffbfeef397b9ecdb57ff8663d5bfb2a8c3e
  Also reproduced on 2026-09-10 nightly (d1fbb1e8c)
- Deployment mode: cluster, 12CU enterprise instance (1 replica), Zilliz Cloud staging (tencent region)
- SDK version: milvus-java-sdk (custom test tool)
- OS: Linux

Current Behavior

During a 10-hour stability test that continuously deletes and rewrites 1M rows with concurrent search, search requests are cancelled mid-execution and fail with user-visible errors. The failures happen on both disk (cardinal/DiskANN) and in-memory (faiss IVF) indexes, i.e. it is not specific to one index type.

Three error classes repeat throughout the run (timestamps UTC, instance in01-899be8ce0166191, 2026-09-16):

1. Search cancelled — cardinal (disk index):

[2026/09/16 15:32:44.034 +00:00] [ERROR] [CGO] [exec/Driver.cpp:427]
  ["[SERVER][RunInternal][MILVUS_SEARCH_3][]Operator::GetOutput failed for [Operator:PhyVectorSearchNode, plan node id: 8948733] :  => failed to search"]
[2026/09/16 15:32:44.034 +00:00] [ERROR] [CGO] [index/index_impl.cpp:82]
  ["[CARDINAL][Search][MILVUS_SEARCH_3] cardinal inner error: Future was cancelled"]
 => failed to search: cardinal inner error: Cardinal search failed at /root/milvus/internal/core/src/index/VectorDiskIndex.cpp:794

(repeated at 16:21, 16:23, 17:13, 20:35, 20:36)

2. Search cancelled — faiss IVF (memory index), also observed:

[2026/09/16 16:21:12.431 +00:00] [WARN] [CGO] [ivf/ivf.cc:1161]
  ["[KNOWHERE][Search][MILVUS_SEARCH_4] faiss inner error: Future was cancelled"]
config={"k":59,"metric_type":"L2","nprobe":"32","refine_ratio":6,"refine_with_quant":true,...}

3. Delete task fails because delegator is not queryable:

[2026/09/16 15:22:34.392 +00:00] [ERROR] [proxy/impl.go:2659]
  ["Failed to run delete task: failed to search/query delegator 1 for channel ...-rootcoord-dml_0_469122492984460697v0: proxy TaskCondition ..."]
[2026/09/16 15:22:34.392 +00:00] [WARN] [proxy/task_delete.go:651]
  ["fail to execute complex delete"] [deleteCnt=275000] [interval=801.184077ms]

The error bursts align with the delete→insert→flush→compact cycle of the test (compaction / segment transitions).

Expected Behavior

Search should be transparent to segment lifecycle transitions (seal/compact/release). Cancelling in-flight search futures during segment churn surfaces as client-visible failures — the test reports 5–8 abnormal rounds out of 12 loops, each round = real SDK errors returned to the client.

Steps To Reproduce

Stability template (QTP id 300 "[stability][v3.0]--perf-12cu", runs daily):

Create 12CU cluster instance (v3.0 nightly)
Collection: Int64 PK, 128d FloatVector + SparseFloatVector(BM25) + 2 VarChar + 2 JSON, 4M rows, BM25 function; indexes: FloatVector AUTOINDEX/L2, SparseFloatVector AUTOINDEX/BM25, VarChar TRIE, JSON INVERTED
Load → Flush → Compact → wait 15min
Concurrent loop (10h): Dense Search (nq=10, topK=100) + Sparse Search (nq=10, topK=100) + per-loop Delete 1M (Int64 < 1000000) → wait 5min → Insert 1M (startId=0) → Flush → Compact → wait 30min → Query count(*)

Anything else?

Reproduced on two different instances/days with the same signature:
- 2026-09-10: instance in01-89f9be8f9f2dc34 (10 ERROR lines / 8 abnormal rounds), incl. At LoadSegment: ChunkedSegmentSealedImpl::ApplyLoadDiff() cancelled ... segcore Future was cancelled
- 2026-09-16: instance in01-899be8ce0166191 (10 ERROR lines / 5 abnormal rounds)
The instance itself stays healthy otherwise: no panic, no restart, no OOM in the run window.
Related but distinct: #52705 (DEADLINE_EXCEEDED), #50554 (delegator subtask stuck) — this one is specifically search futures being cancelled during segment transitions.