[Bug]: StreamingNode flusher panics on an unknown V2 broadcast message type and crash-loops on WAL replay
Environment
- Milvus Version: master (observed against
2cbfc53b81; the pattern applies to every release that added a new V2 broadcast message type) - Deployment Mode: cluster
- MQ: any (streaming WAL; reproduced by reasoning over the WAL path, not MQ-specific)
- SDK: n/a
- OS: any
Reproduction
Option B: Steps (systemic / rolling-upgrade issue)
- Deploy: a cluster whose streamingnodes run a build older than a new V2 broadcast message type (for a concrete case: newer than 3.0 dev before the two-phase import work, i.e. before #53544), with DataCoord from a newer build.
- Setup: any collection with a primary key.
- Workload: submit an ordinary bulk import (any non-backup / non-L0 import; the two-phase import work triggers
AssigningIDRangefor all of them, not only autoID). - Trigger: the new DataCoord broadcasts
ImportIDRange(V2,MessageType49) to the job's data vchannels. - Observed via: the old streamingnode's process dies with a panic; after restart the WAL is replayed from the checkpoint, the same message is converted again, and it dies again.
Trigger Conditions
- Frequency: always, deterministically, for the affected pchannel.
- First observed after: the first message of the new V2 type delivered on a data vchannel while an old streamingnode is serving.
- Does NOT happen when: the message is broadcast control-channel-only (the flusher returns early on
IsControlChannel && !IsPChannelLevel), or when all streamingnodes are upgraded, or when ordinary imports are drained/paused during the upgrade window.
Expected Behavior
An unknown message type must not kill a reader process. The flusher should handle a type it does not know with a controlled outcome - a WARN + no-op, the same pattern the flusher already uses for message types it deliberately ignores - instead of a panic, so a rolling upgrade cannot take down every other message on the pchannel.
Actual Behavior
WALFlusherImpl.dispatch has no case for the new V2 type, so the message falls through to flusherComponents.HandleMessage -> dataSyncServiceWrapper.HandleMessage -> handler.GenerateMsgPack -> addMsgPackIntoPending -> NewMsgPackFromMessage -> parseSingleMsgPayload -> fromMessageToTsMsgV2, whose type switch reaches default: panic("unsupported message type") (pkg/streaming/util/message/adaptor/message.go:171; handler.go:148 has the same shape for an unsupported message version).
Nothing on that path recovers: addMsgPackIntoPending only logs the returned error (a panic is not an error), dispatch's deferred func only records the time tick, and the consume loop only handles errors. The panic therefore reaches the process. It also bypasses the flusher's normal fatal notification path (dispatch returning an error -> notifyFatal).
Because the WAL replays from the checkpoint after the crash, the same message is converted again, so the node crash-loops until it is upgraded. While it loops, every other message on that pchannel (DML, other DDL) stops too, so the blast radius is larger than the import itself.
Error Logs
panic: unsupported message type
github.com/milvus-io/milvus/pkg/v3/streaming/util/message/adaptor.fromMessageToTsMsgV2(...)
pkg/streaming/util/message/adaptor/message.go:171
...parseSingleMsgPayload -> NewMsgPackFromMessage -> addMsgPackIntoPending -> GenerateMsgPack
...dataSyncServiceWrapper.HandleMessage -> flusherComponents.HandleMessage -> WALFlusherImpl.dispatch
Non-default Configuration
None. The window is created purely by component upgrade order.
Analysis Hints (Optional)
- Suspect code location:
pkg/streaming/util/message/adaptor/message.go(fromMessageToTsMsgV2default branch andhandler.goversion branch). - Why it is the flusher specifically: the flusher path is the only production caller of
NewMsgPackFromMessage(V1 conversion dispatches throughcommonpb.MsgType, which go-api knows; the V2 conversion is a hardcoded type table with a panicking default). DataCoord/streamingcoord callers of the same layer only fail the broadcast and retry, which is fail-safe. - Precedent:
CommitImport/RollbackImportavoid this because their explicit flusher cases shipped in the same release as the message type. Every new V2 broadcast type is a landmine for older readers until this is made controlled. - Related PR (introduces the concrete new type): #53544. Today's mitigation is documented there and in the CDC import user guide: upgrade the streaming nodes before DataCoord, or drain/pause ordinary imports for the window.
- Already ruled out: not a DataCoord bug and not MQ-specific.
Source: milvus-io/milvus