[Bug]: New Workers break StorageV3 manifest compatibility with 3.0 DataCoord
Is there an existing issue for this?
- I have searched the existing issues.
Environment
- Reported version pairing: DataCoord from the Milvus
3.0branch, Worker from currentmasterafter #52621. - Source versions used for the compatibility audit:
3.0@9b02c9e04431f38ece9bdf42a0b3b0bbbb22fccfandmaster@cd06d59356c5c26e9166663fea1de835ef74dd44. - Storage format: StorageV3.
- Exact deployment image tags and other environment details were not provided.
Current Behavior
During a rolling upgrade with newer Workers and an older DataCoord, a Worker successfully generates the data for newly added fields but returns ManifestDelta with an empty Manifest. The older DataCoord requires a complete StorageV3 manifest and repeatedly rejects the schema-bump compaction result, preventing metadata publication and task completion.
The protocol audit found two related paths changed by #52621: standalone text-index and JSON-stats jobs now leave manifest commits to DataCoord. An older DataCoord instead expects the returned manifest to already contain those stats, so the generated stats are absent from the returned manifest.
Expected Behavior
New Workers should preserve the legacy complete-manifest response contract for requests from old coordinators. Coordinator-side manifest commits should only be used when the requesting coordinator explicitly advertises support. Sort jobs should continue returning a complete manifest.
Steps To Reproduce
- Use a DataCoord from the
3.0branch with Workers built frommasterafter #52621. - Prepare a collection with existing sealed StorageV3 segments.
- Add a field that requires physical backfill and allow the schema-bump compaction task to run on a new Worker.
- Observe that data generation succeeds, but DataCoord repeatedly fails to save the compaction result because its complete-manifest field is empty.
These steps describe the reported mixed-version failure. Local regression tests additionally reproduced all three legacy-request failures against the unpatched Worker source; a full mixed-binary cluster run has not been performed as part of the fix.
Milvus Log
bumpSchemaVersionTask failed to save segment meta
schema bump compaction result should contain a StorageV3 manifest:
compaction plan illegal
Analysis and Proposed Fix
Introduced by #52621. Adding protobuf fields did not preserve the old response semantics when manifest transaction ownership moved from Worker to DataCoord.
Proposed fix: #53552 adds a per-request capability, defaulting to the legacy contract when omitted. It covers additive schema bump, standalone text-index jobs, and standalone JSON-stats jobs, with real-manifest regression tests and metadata replay coverage.
This is a separate compatibility regression from #51723, which tracks manifest CAS and metadata-lock issues.
Source: milvus-io/milvus