[Enhancement]: CDC failover: recover the stale primary as a new standby (wipe + backup restore + reverse CDC)
What would you like to be added?
After force_promote, treat the original primary as stale / forked. Provide a first-class, documented, and e2e-tested way to recover it as a new standby of the promoted cluster, using:
- wipe the old primary
milvus-backuprestore from the new primary- reverse CDC:
new primary → old primary
End state: app writes → B (primary) --B's CDC--> A (standby, reject writes). A is fenced; B is the only writer.
1. Wipe — add milvus wipe-storage
There is no supported wipe today:
drop collectionis not enough (replicate config, CollectionIDs, dirty WAL, object binlogs remain)milvus mckonly checks QueryCoord task consistency; it does not delete business data
Add a CLI on the main binary (cmd/milvus/), next to run / stop / mck: --dry-run and --execute are mutually exclusive. Omit both → dry-run. No interactive Y/n (must run unattended in a Job).
Safety: before touching any store, read {etcd.metaRoot}/session/ (same ETCD_ROOTPATH). If any component session exists (mixcoord including standby, proxy, querynode, datanode, streamingnode, cdc), refuse both dry-run and --execute. Ignore allocator keys session/id and session/version. No --force. Caller stops the cluster first; leftover session keys mean it is still running.
Three prefixes are independent:
- etcd:
etcd.rootPath(ETCD_ROOTPATH) → delete the entire prefix (meta+kv) - Kafka:
msgChannel.chanNamePrefix.cluster(KAFKA_MSG_CHANNEL_CLUSTER) → delete topics named{prefix}-*(do not truncate-and-keep) - object storage:
minio.bucketName+minio.rootPath→ if rootPath is non-empty,RemoveWithPrefix(rootPath); if empty, delete only Milvus data dirs (insert_log/,delta_log/,stats_log/,bm25_stats/,index_files/,text_log/,json_stats/,json_key_index_log/,part_stats/,analyze_stats/), assuming the bucket is exclusive to this cluster
Empty etcd or Kafka prefix after trim → refuse (including dry-run). Do not require the three prefixes to be equal.
Order: connect → session check → print inventory → if not --execute, exit 0 → object storage → Kafka → etcd last. Missing targets = success (idempotent). No auto-rollback. Log every prefix / topic / dir; do not print only wipe done.
Dry-run inventory:
- etcd: collection count; per collection name / id / segment count / row count (
NumOfRowssum, including Dropped segments still in meta) - Kafka: cluster prefix + topic list
- object storage: file counts per dir via
WalkWithPrefix(no sizes)
Why is this needed?
force_promote forks the old primary. Re-attaching it as incremental standby is unsafe. Promote exists (#47351); wiping the old primary does not. Without wipe, restore stacks on dirty meta/WAL/objects.
Scope
- Phase 1 (this PR):
milvus wipe-storage - Later: backup restore + reverse CDC as the documented/e2e recovery loop
- Not in phase 1: KubeBlocks Ops/UI, milvusctl, stuffing this into
mck, deleting the Kafka/etcd cluster Config: paramtable only (user.yaml+ native env), same as mixcoord. Do not take--etcdIp/--minioAddressas source of truth. Process must not register a session, steal MixCoord, or write business WAL.
Source: milvus-io/milvus