#53512·milvus

[Enhancement]: CDC failover: recover the stale primary as a new standby (wipe + backup restore + reverse CDC)

Author: sunwshCreated Sep 16, 2026Updated Sep 17, 2026
Labelskind/enhancement

What would you like to be added?

After force_promote, treat the original primary as stale / forked. Provide a first-class, documented, and e2e-tested way to recover it as a new standby of the promoted cluster, using:

  1. wipe the old primary
  2. milvus-backup restore from the new primary
  3. reverse CDC: new primary → old primary

End state: app writes → B (primary) --B's CDC--> A (standby, reject writes). A is fenced; B is the only writer.

1. Wipe — add milvus wipe-storage

There is no supported wipe today:

  • drop collection is not enough (replicate config, CollectionIDs, dirty WAL, object binlogs remain)
  • milvus mck only checks QueryCoord task consistency; it does not delete business data

Add a CLI on the main binary (cmd/milvus/), next to run / stop / mck: --dry-run and --execute are mutually exclusive. Omit both → dry-run. No interactive Y/n (must run unattended in a Job).

Safety: before touching any store, read {etcd.metaRoot}/session/ (same ETCD_ROOTPATH). If any component session exists (mixcoord including standby, proxy, querynode, datanode, streamingnode, cdc), refuse both dry-run and --execute. Ignore allocator keys session/id and session/version. No --force. Caller stops the cluster first; leftover session keys mean it is still running.

Three prefixes are independent:

  • etcd: etcd.rootPath (ETCD_ROOTPATH) → delete the entire prefix (meta + kv)
  • Kafka: msgChannel.chanNamePrefix.cluster (KAFKA_MSG_CHANNEL_CLUSTER) → delete topics named {prefix}-* (do not truncate-and-keep)
  • object storage: minio.bucketName + minio.rootPath → if rootPath is non-empty, RemoveWithPrefix(rootPath); if empty, delete only Milvus data dirs (insert_log/, delta_log/, stats_log/, bm25_stats/, index_files/, text_log/, json_stats/, json_key_index_log/, part_stats/, analyze_stats/), assuming the bucket is exclusive to this cluster

Empty etcd or Kafka prefix after trim → refuse (including dry-run). Do not require the three prefixes to be equal.

Order: connect → session check → print inventory → if not --execute, exit 0 → object storage → Kafka → etcd last. Missing targets = success (idempotent). No auto-rollback. Log every prefix / topic / dir; do not print only wipe done.

Dry-run inventory:

  • etcd: collection count; per collection name / id / segment count / row count (NumOfRows sum, including Dropped segments still in meta)
  • Kafka: cluster prefix + topic list
  • object storage: file counts per dir via WalkWithPrefix (no sizes)

Why is this needed?

force_promote forks the old primary. Re-attaching it as incremental standby is unsafe. Promote exists (#47351); wiping the old primary does not. Without wipe, restore stacks on dirty meta/WAL/objects.

Scope

  • Phase 1 (this PR): milvus wipe-storage
  • Later: backup restore + reverse CDC as the documented/e2e recovery loop
  • Not in phase 1: KubeBlocks Ops/UI, milvusctl, stuffing this into mck, deleting the Kafka/etcd cluster Config: paramtable only (user.yaml + native env), same as mixcoord. Do not take --etcdIp / --minioAddress as source of truth. Process must not register a session, steal MixCoord, or write business WAL.