volume.scrub does not report a damaged header (cookie/id) on a live needle, while reads of it return 404

Author: JeremyMcdrCreated Sep 25, 2026Updated Sep 25, 2026

Describe the bug

If the first 12 bytes of a live needle (cookie + id) are damaged on disk, none of the volume.scrub modes (index, local, full, reads) reports the volume. Yet every read of that needle on that replica returns 404.

In a replicated cluster, reads fall back to another replica, so nothing fails visibly. The volume silently has one healthy replica fewer, and every later scrub still reports it clean. On a replicated cluster, volume.check.disk did not report it either.

With only the 8 id bytes damaged, reads still return 200, since the cookie still matches, and scrub is still silent.

Likely cause

scrubVolumeData (volume_checking.go#L69-L107) reads each live needle with n.ReadData. That validates the data CRC, which does not cover the header, but the header id is never compared with the index key. On master, #11396 added that comparison for deleted needles only (volume_checking.go#L95). Comparing n.Id with the index key for live needles too would catch any header damage that touches the id. The cookie cannot be checked from the index alone.

System Setup

chrislusf/seaweedfs:4.47 — weed version: version 30GB 4.47 c50733600 linux arm64
weed server -dir=/data -volume.max=5 in a single container (Docker 29.2, linux/arm64)

Steps to reproduce

bash

#!/usr/bin/env bash
# SeaweedFS 4.47: volume.scrub does not report a live needle whose header (cookie + id) is damaged.
IMG=${IMG:-chrislusf/seaweedfs:4.47}
docker run -d --name sw "$IMG" server -dir=/data -volume.max=5 >/dev/null
IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' sw)
x() { docker exec sw sh -c "$1"; }
until x "wget -qO- http://$IP:8080/status" >/dev/null 2>&1; do sleep 1; done; sleep 3
# byte offset of a needle, read from the volume's .idx (key 8B | offset 4B, in 8-byte units | size 4B)
offset_of() {
  local vid=${1%%,*} rest=${1#*,}; local key=$((16#${rest:0:${#rest}-8}))
  x "od -An -tx1 -v -w16 /data/$vid.idx" | tr -d ' ' | while read -r l; do
    [ ${#l} -eq 32 ] && [ $((16#${l:0:16})) -eq $key ] && echo $((16#${l:16:8} * 8))
  done | tail -1
}
upload() { x "weed upload -master=localhost:9333 $1" | sed -n 's/.*"fid":"\([^"]*\)".*/\1/p'; }
x 'head -c 300000 /dev/urandom > /tmp/f'
fid=$(upload /tmp/f); vid=${fid%%,*}; off=$(offset_of "$fid")
echo "== fid $fid, needle at byte $off of volume $vid"
x "wget -S -O /dev/null http://$IP:8080/$fid 2>&1 | grep HTTP/"
# the first 12 bytes of the needle: cookie (4) + id (8)
x "dd if=/dev/urandom of=/data/$vid.dat bs=1 count=12 seek=$off conv=notrunc 2>/dev/null"
for mode in index local full reads; do
  echo "== volume.scrub -mode $mode"
  x "printf 'lock\nvolume.scrub -volumeId $vid -node $IP:8080 -mode $mode\nunlock\n' | weed shell -master=localhost:9333 2>&1 | grep -E 'Scrubbed|failures|Affected'"
done
x "wget -S -O /dev/null http://$IP:8080/$fid 2>&1 | grep HTTP/"
docker rm -f -v sw >/dev/null

Output on 4.47:

bash

== fid 3,01f76a1927, needle at byte 8 of volume 3
  HTTP/1.1 200 OK
== volume.scrub -mode index
Scrubbed 1 files and 1 volumes on 1 nodes
== volume.scrub -mode local
Scrubbed 1 files and 1 volumes on 1 nodes
== volume.scrub -mode full
Scrubbed 1 files and 1 volumes on 1 nodes
== volume.scrub -mode reads
Scrubbed 1 files and 1 volumes on 1 nodes
  HTTP/1.1 404 Not Found
wget: server returned error: HTTP/1.1 404 Not Found

Expected behavior

At least volume.scrub -mode full reports the volume: the needle at the indexed offset does not carry the indexed id.

Source: seaweedfs/seaweedfs

View original on GitHub View discussion on GitHub