[Bug]: 2.6 scalar index reload can starve old-index deletion
Environment
Milvus 2.6: inspected hotfix-2.6.23 (ec77bc31e) and the 2.6 branch at d1653e8547.
Current behavior
Replacing a scalar index on a loaded segment can leave the old index resident and repeatedly attempt to load the replacement.
IndexChecker.checkReplica generates missing-index load tasks before redundant-index drop tasks. Both use low priority and the scheduler conflict key (replicaID, segmentID, isGrowing). When the load task remains registered, the following drop is rejected before RPC dispatch. The old scalar index can then prevent the new index from attaching. Later checks can repeat the same ordering.
A failed DropIndex RPC is not required: the replacement can become ready before asynchronous deletion has completed. A stale cached worker is an additional possible trigger, because delegator DropIndex traverses all cached workers and returns on the first error.
Two related implementation gaps:
- DropIndexTask checks the segment task index during admission but is not registered or removed there, allowing a later same-segment load or duplicate drop to enter.
- QueryNode.DropIndex ignores errors returned by the local segment and can report success when deletion failed.
Expected behavior
Deletion and replacement loading should make progress in dependency order. Deletion should target the relevant segment holder, and local failures must reach the coordinator. Successful partial deletion must remain visible in distribution reports.
Reproduction and evidence
White-box setup:
- A replica reports scalar index A on a sealed segment. Desired metadata contains replacement B for the same field, and B's index files are ready.
- Run the index checker: it returns Load B and Drop A for that segment.
- Add Load B, then Drop A before the load task is removed: Drop A is rejected with
task with the same segment exists. - Leave A reported and repeat after removing the failed load task: the same admission result repeats.
Separately, adding Drop A before Load B admits both tasks because drop registration is missing. Injecting a local segment deletion error also demonstrates the QN handler returning success.
These behaviors were exercised with extracted production function bodies and minimal dependency stubs. This is not a live-cluster reproduction. Native package tests are currently blocked locally by unavailable milvus_core/rocksdb libraries and a Go 1.26/sonic compatibility problem.
Related work
Master changed index reconciliation to LoadDiff-based reopen in #48206. The 2.6 branch still uses the separate load/drop path described here.
Source: milvus-io/milvus