Volume server: a needle over 1 MiB whose data fails its CRC is served as 200 with the corrupted bytes (error only logged)

Author: JeremyMcdrCreated Sep 25, 2026Updated Sep 25, 2026

Describe the bug

When the data of a needle larger than 1 MiB is damaged on disk, a GET to the volume server returns 200 OK with the full Content-Length and the corrupted bytes. The CRC mismatch is detected, but only after the body has been sent, and it only appears in the volume server log:

E0925 14:41:03.555056 common.go:308 ProcessRangeRequest: ReadNeedleData checksum 3492983253 expected 3947339564 for Needle: 1,016857dd95 Size:2097172, DataSize:2097152, Name:f2097152, Mime: Compressed:false

The same damage on a 64 KiB needle is correctly answered with 500, so a reader of a replicated volume (filer, S3 gateway) can fall back to another replica. Above 1 MiB it cannot: the response looks complete and valid.

We also observed this through the S3 gateway on a replicated cluster (4 volume servers, replication 101), with one replica damaged. On one run, the gateway kept reading that replica: 30 reads out of 30 returned the corrupted bytes with 200, until volume.scrub and repair. The AWS SDK v3 rejected them with its response checksum. A client that does not validate checksums, such as a browser following a presigned URL, gets wrong data silently. We measured the threshold between 1,048,000 bytes (500, fallback) and 1,048,575 bytes (200, corrupted), which matches PagedReadLimit. With -encryptVolumeData, the same damage makes the read fail instead, since decryption is authenticated.

Likely cause

Above PagedReadLimit (volume_read.go#L18), the needle is read meta-only and streamed by readNeedleDataInto. The CRC is only compared after the copy loop (volume_read.go#L217-L222), once the status line and the bytes have already gone out through ProcessRangeRequest. The http.Error(w, …, 500) there (common.go#L307-L311) can no longer change the response. Ranged reads of a needle above that limit are not checked at all, since the check requires offset == 0 && size == DataSize.

A possible direction: hold back the last page until the CRC is verified, and abort the connection on a mismatch (panic(http.ErrAbortHandler)). The client then sees a failed transfer and can retry another replica.

System Setup

chrislusf/seaweedfs:4.47 — weed version: version 30GB 4.47 c50733600 linux arm64
weed server -dir=/data -volume.max=5 in a single container (Docker 29.2, linux/arm64)

Steps to reproduce

bash

#!/usr/bin/env bash
# SeaweedFS 4.47: a needle > 1 MiB whose data is damaged is served as 200 with the corrupted bytes.
IMG=${IMG:-chrislusf/seaweedfs:4.47}
docker run -d --name sw "$IMG" server -dir=/data -volume.max=5 >/dev/null
IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' sw)
x() { docker exec sw sh -c "$1"; }
until x "wget -qO- http://$IP:8080/status" >/dev/null 2>&1; do sleep 1; done; sleep 3
# byte offset of a needle, read from the volume's .idx (key 8B | offset 4B, in 8-byte units | size 4B)
offset_of() {
  local vid=${1%%,*} rest=${1#*,}; local key=$((16#${rest:0:${#rest}-8}))
  x "od -An -tx1 -v -w16 /data/$vid.idx" | tr -d ' ' | while read -r l; do
    [ ${#l} -eq 32 ] && [ $((16#${l:0:16})) -eq $key ] && echo $((16#${l:16:8} * 8))
  done | tail -1
}
upload() { x "weed upload -master=localhost:9333 $1" | sed -n 's/.*"fid":"\([^"]*\)".*/\1/p'; }
for size in 2097152 65536; do
  x "head -c $size /dev/urandom > /tmp/f$size"
  fid=$(upload /tmp/f$size); off=$(offset_of "$fid")
  # needle (version 3): cookie 4 | id 8 | size 4 | DataSize 4 | data -> damage 16 bytes, 1000 bytes into the data
  x "dd if=/dev/urandom of=/data/${fid%%,*}.dat bs=1 count=16 seek=$((off + 20 + 1000)) conv=notrunc 2>/dev/null"
  echo "== $size bytes, fid $fid"
  x "rm -f /tmp/out; wget -S -O /tmp/out http://$IP:8080/$fid 2>&1 | grep -E 'HTTP/|Content-Length'"
  x "sha256sum /tmp/f$size /tmp/out 2>/dev/null"
done
docker logs sw 2>&1 | grep 'ReadNeedleData checksum'
docker rm -f -v sw >/dev/null

Output on 4.47:

bash

== 2097152 bytes, fid 1,016857dd95
  HTTP/1.1 200 OK
  Content-Length: 2097152
3edacb5fd9edb9881cc561191cf3889a195fbdc242f2b850ce3e6c9a4b223a26  /tmp/f2097152
2d5057e2ee5f4829dfd756345c2a8845d8b70aabc160847a5d6e377dce67467d  /tmp/out
== 65536 bytes, fid 2,02de15f671
  HTTP/1.1 500 Internal Server Error
wget: server returned error: HTTP/1.1 500 Internal Server Error
4a0b2c6fbcf922113fb5c19293dfe389235f443188e84feffffe65ea98808bff  /tmp/f65536
E0925 14:41:03.555056 common.go:308 ProcessRangeRequest: ReadNeedleData checksum 3492983253 expected 3947339564 for Needle: 1,016857dd95 Size:2097172, DataSize:2097152, Name:f2097152, Mime: Compressed:false

Expected behavior

A needle that fails its CRC is never delivered as a successful response, whatever its size. The reader must be able to see the failure and fall back to another replica.

Source: seaweedfs/seaweedfs

View original on GitHub View discussion on GitHub