[Enhancement]: Assign import ID ranges after preimport via a two-phase ImportIDRange message
Author: xaxysCreated Sep 17, 2026Updated Sep 17, 2026
Is there an existing issue for this?
- I have searched the existing issues
What would you like to be added?
Replace the broadcast-time PK range pre-allocation for bulk import (#51667 / #51825) with a two-phase flow:
ImportV2broadcastsImportMsgonly: no file sizing (parquet footer / npy header reads / byte-derived bounds), no ID allocation, no per-file range on the message, so the user-facing RPC no longer does object-store I/O proportional to the file count.- After the PreImport tasks report exact per-file row counts, the cluster acting as primary allocates one contiguous ID range per file, sized exactly to that file's row count, and broadcasts a new
ImportIDRangeWAL message to the job's data vchannels plus the control channel. - Every cluster's ack callback applies the ranges to its local import job meta. The job waits in a new
ImportJobState.AssigningIDRangeuntil the ranges are present, then proceeds toImporting. - The per-file range becomes the datanode's PK (autoID) / RowID (explicit PK) cursor, so the primary and every CDC secondary derive identical PK/RowID.
- Because the range is sized from the exact post-preimport count, the range size doubles as the cross-cluster divergence authority: the range gate fails the job when a cluster's local preimport count differs from the range size, before any segment is written.
Why is this needed?
- The current design makes the user-facing
ImportV2RPC read every file's footer/header (or send a HEAD per object) before returning, so its latency scales with the number of files and can reach seconds to minutes for large multi-file imports. - Sizing at broadcast time can only produce an upper bound for formats without an exact recorded count (JSON/CSV), so the reservation over-allocates and the exact count is only settled later, at assemble time.
- The ranges travel on
ImportMsg, which is broadcast at job creation - before any row count exists. Moving them to a dedicated post-preimport message keeps the exact count and the range on the same message, and leaves one explicit place (the range gate) where cross-cluster divergence is detected and fails the job loudly.
Anything else?
- Design doc:
docs/design-docs/design_docs/cdc/20260915-import-id-range-two-phase.md(will be submitted in a separate doc PR). - Implementation PR: see the comment below.
Source: milvus-io/milvus