#50915 [Bug]: StreamingNode panics when Vortex growing flush close/upload fails with XAmzContentChecksumMismatch

Environment

Milvus Version: harbor.milvus.io/milvusdb/milvus:master-20260628-4018b294
Deployment Mode: cluster, Milvus Operator
Instance / Namespace: qa-milvus/laion1b-test-2
MQ / WAL: streaming mode, Pulsar WAL
Object Storage: MinIO/S3-compatible backend, bucket bucket-laion2
Workload: mixed read/write LAION 1B benchmark
Workflow: https://argo-workflows.zilliz.cc/workflows/qa/laion1b-test-cron-sn-nullable-1782741600?tab=workflow
Failed pod: laion1b-test-2-milvus-streamingnode-7f796c8cfc-2h8rq

Kubernetes termination status:

restartCount=1
lastReason=Error
lastExitCode=134
lastStartedAt=2026-06-29T08:50:41Z
lastFinishedAt=2026-06-29T21:09:23Z

Reproduction

Steps

Deploy Milvus cluster with StreamingNode enabled, using image master-20260628-4018b294.
Enable StorageV3 / Vortex:

common:
  enabledJSONShredding: false
  storage:
    useLoonFFI: true
dataNode:
  storage:
    format: vortex

Use a large LAION-style collection (laion_stable_10_nullable) with continuous mixed traffic.
Run the mixed read/write workflow for 7h:
- 20 concurrent users
- task weights load:query:search:upsert = 1:4:13:2
- search: nq=1, top_k=100, timeout 10s
- query: count(*)
- upsert: numRows=500, random id/vector, timeout 60s
During the run, StreamingNode flushes growing segment data through the Vortex writer path.
A Vortex object close/upload fails with XAmzContentChecksumMismatch.
StreamingNode panics and restarts with exit code 134. Search/query then see channel unavailable.

Trigger Conditions

Frequency: observed once in this latest laion1b workflow after several hours of mixed DQL/DML traffic.
First observed in this run: 2026-06-29T21:09:22Z
Direct trigger: FlushGrowingSegmentData fails while closing/uploading a Vortex file.
Important distinction: the object storage checksum mismatch is the trigger, but the Milvus bug is that the returned storage error is converted into a process-fatal panic in StreamingNode.

Expected Behavior

A transient or retryable object storage write/close failure while flushing growing source data should not abort the entire StreamingNode process.

Expected behavior should be one of:

retry/backoff the failed Vortex object upload;
fail the affected sync task and keep the process alive;
mark the affected channel/segment unhealthy and recover through coordination;
surface an error without crashing StreamingNode.

The process should not exit with panic(err) just because FlushGrowingSegmentData returned an I/O error.

Actual Behavior

StreamingNode panics and exits with code 134:

[2026/06/29 21:09:22.688 +00:00] [WARN] [CGO] [s3/s3_internal.h:194]
AWS Error UNKNOWN (HTTP status 400) during PutObject operation:
Unable to parse ExceptionName: XAmzContentChecksumMismatch
Message: The provided 'x-amz-checksum' header does not match what was computed.

[2026/06/29 21:09:22.690 +00:00] [WARN] [segments/segment.go:1782] ["CStatus returns err"]
error="IOError: Failed to close Vortex file: Failed to close ObjectStoreWriterCpp:
ERROR: Internal error(code 3) details: IOError: When uploading object with key
'laion1b-test-2/insert_log/467096650944088381/467096650944088397/467330142211798141/_data/0_1f3d1be4-5340-4c30-b068-55179c274859.vortex'
in bucket 'bucket-laion2': AWS Error UNKNOWN (HTTP status 400) during PutObject operation:
Unable to parse ExceptionName: XAmzContentChecksumMismatch ..."

panic: flush growing source data: IOError: Failed to close Vortex file:
Failed to close ObjectStoreWriterCpp: ERROR: Internal error(code 3) details:
IOError: When uploading object with key '.../_data/0_1f3d1be4-5340-4c30-b068-55179c274859.vortex'
...

After the StreamingNode restart, client-side query/search failures appear:

no available shard leaders: channel not available[channel=laion1b-test-2-rootcoord-dml_1_467096650944088381v1]
Deadline Exceeded
proxy TaskCondition context Done: context deadline exceeded

Client log timing:

rate=0 upsert throttling window: 2026-06-29 20:31-21:00 UTC
StreamingNode panic: 2026-06-29 21:09:22-21:09:23 UTC
channel not available: starts around 2026-06-29 21:09 UTC
Search deadline exceeded: 2026-06-29 21:11 UTC

Error Logs

Milvus Loki link for developers:

https://grafana-4am.zilliz.cc/explore?schemaVersion=1&panes=%7B%22snpanic%22%3A%7B%22datasource%22%3A%22c2fa6d21-2b11-43ec-921f-ffefca84f260%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22%7Bnamespace%3D%5C%22qa-milvus%5C%22%2C%20pod%3D~%5C%22laion1b-test-2-milvus-streamingnode-7f796c8cfc-2h8rq%5C%22%7D%20%7C~%20%5C%22XAmzContentChecksumMismatch%7CFailed%20to%20close%20Vortex%20file%7Cpanic%3A%20flush%20growing%20source%20data%7CFlushGrowingSegmentData%7CSIGABRT%7CERROR%5C%22%22%2C%22queryType%22%3A%22range%22%2C%22datasource%22%3A%7B%22type%22%3A%22loki%22%2C%22uid%22%3A%22c2fa6d21-2b11-43ec-921f-ffefca84f260%22%7D%2C%22editorMode%22%3A%22code%22%2C%22direction%22%3A%22forward%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%222026-06-29T21%3A08%3A30.000Z%22%2C%22to%22%3A%222026-06-29T21%3A10%3A30.000Z%22%7D%7D%7D&orgId=1

Relevant stack frames from the panic:

github.com/milvus-io/milvus/internal/querynodev2/segments.HandleCStatus
  /go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/cgo_util.go:49
github.com/milvus-io/milvus/internal/querynodev2/segments.(*LocalSegment).FlushData
  /go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment.go:1782
github.com/milvus-io/milvus/internal/querynodev2/delegator.(*delegatorGrowingFlushSource).FlushGrowingData
  /go/src/github.com/milvus-io/milvus/internal/querynodev2/delegator/growing_flush_source.go:547
github.com/milvus-io/milvus/internal/flushcommon/syncmgr.(*GrowingSourceSyncTask).Run
  /go/src/github.com/milvus-io/milvus/internal/flushcommon/syncmgr/growing_source.go:576
github.com/milvus-io/milvus/internal/flushcommon/syncmgr.(*GrowingSourceSyncTask).HandleError
  /go/src/github.com/milvus-io/milvus/internal/flushcommon/syncmgr/growing_source.go:510
github.com/milvus-io/milvus/internal/flushcommon/writebuffer.defaultWBOption.func1
  /go/src/github.com/milvus-io/milvus/internal/flushcommon/writebuffer/options.go:47

Non-default Configuration

Relevant Milvus CR config:

{
  "common": {
    "enabledJSONShredding": false,
    "storage": {
      "useLoonFFI": true
    }
  },
  "dataNode": {
    "storage": {
      "format": "vortex"
    },
    "dataSync": {
      "maxParallelSyncTaskNum": 64
    }
  },
  "dataCoord": {
    "segment": {
      "maxSize": 4096,
      "sealProportion": 0.08,
      "expansionRate": 1.15
    }
  },
  "rootCoord": {
    "dmlChannelNum": 16
  },
  "quotaAndLimits": {
    "dml": {
      "enabled": true,
      "deleteRate": {
        "max": 0.5
      }
    }
  }
}

Analysis Hints

This looks like two issues chained together:

Storage layer trigger: Vortex writer / ObjectStoreWriterCpp / S3 path gets XAmzContentChecksumMismatch while closing/uploading a .vortex file. This is close to milvus-storage#488, but that issue mentions loon manifest writes; this failure is a Vortex data file path under insert_log/.../_data/*.vortex.
Milvus fatal error handling: the returned error is propagated through growing-source flush and then converted into a process panic:

// internal/querynodev2/segments/segment.go
status := C.FlushGrowingSegmentData(...)
if err := HandleCStatus(ctx, &status, "FlushGrowingSegmentData"); err != nil {
    return nil, err
}

// internal/flushcommon/syncmgr/growing_source.go
result, err := t.source.FlushGrowingData(...)
if err != nil {
    return errors.Wrap(err, "flush growing source data")
}
defer func() {
    if err != nil {
        t.HandleError(err)
    }
}()

// internal/flushcommon/writebuffer/options.go
// default error handler, just panicking
errorHandler: func(err error) {
    panic(err)
}

This is related to, but not fully covered by:

https://github.com/milvus-io/milvus/issues/49998 - same general pattern: StreamingNode panics when sync task fails to write object storage; closed as not reproduced recently.
https://github.com/milvus-io/milvus/issues/50884 - Vortex writer panic, but direct root cause is Arrow FFI alignment, not S3 checksum mismatch.
https://github.com/milvus-io/milvus-storage/issues/488 - XAmzContentChecksumMismatch, but reported for loon manifest write; this issue is Vortex data file close/upload plus Milvus process panic.

Why this is a Milvus bug

XAmzContentChecksumMismatch / object storage write failure is an I/O error and should be handled as a recoverable or task-level failure.
The process exits only because Milvus default write buffer error handling calls panic(err).
Kubernetes shows reason=Error, exitCode=134; this was not an OOM kill.
The later channel not available and search/query deadline failures are secondary symptoms after the StreamingNode panic.