#53543·milvus

[Enhancement]: Assign import ID ranges after preimport via a two-phase ImportIDRange message

Author: xaxysCreated Sep 17, 2026Updated Sep 17, 2026

Is there an existing issue for this?

  • I have searched the existing issues

What would you like to be added?

Replace the broadcast-time PK range pre-allocation for bulk import (#51667 / #51825) with a two-phase flow:

  • ImportV2 broadcasts ImportMsg only: no file sizing (parquet footer / npy header reads / byte-derived bounds), no ID allocation, no per-file range on the message, so the user-facing RPC no longer does object-store I/O proportional to the file count.
  • After the PreImport tasks report exact per-file row counts, the cluster acting as primary allocates one contiguous ID range per file, sized exactly to that file's row count, and broadcasts a new ImportIDRange WAL message to the job's data vchannels plus the control channel.
  • Every cluster's ack callback applies the ranges to its local import job meta. The job waits in a new ImportJobState.AssigningIDRange until the ranges are present, then proceeds to Importing.
  • The per-file range becomes the datanode's PK (autoID) / RowID (explicit PK) cursor, so the primary and every CDC secondary derive identical PK/RowID.
  • Because the range is sized from the exact post-preimport count, the range size doubles as the cross-cluster divergence authority: the range gate fails the job when a cluster's local preimport count differs from the range size, before any segment is written.

Why is this needed?

  1. The current design makes the user-facing ImportV2 RPC read every file's footer/header (or send a HEAD per object) before returning, so its latency scales with the number of files and can reach seconds to minutes for large multi-file imports.
  2. Sizing at broadcast time can only produce an upper bound for formats without an exact recorded count (JSON/CSV), so the reservation over-allocates and the exact count is only settled later, at assemble time.
  3. The ranges travel on ImportMsg, which is broadcast at job creation - before any row count exists. Moving them to a dedicated post-preimport message keeps the exact count and the range on the same message, and leaves one explicit place (the range gate) where cross-cluster divergence is detected and fails the job loudly.

Anything else?

  • Design doc: docs/design-docs/design_docs/cdc/20260915-import-id-range-two-phase.md (will be submitted in a separate doc PR).
  • Implementation PR: see the comment below.