Vacuum corrupts 5-byte index offsets during concurrent-write replay; a later vacuum deletes the affected data

Vacuum corrupts 5-byte index offsets during concurrent-write replay; a later vacuum deletes the affected data

Description

In large-disk builds (-tags=5BytesOffset), Volume.makeupDiff() updates only four bytes of an index entry's five-byte offset when replaying writes that arrived after the compaction snapshot.

The fifth byte retains the old offset's high bits. When compaction moves a replayed needle into a different 32 GiB offset range, the resulting index points to the wrong location even though the needle body was copied correctly.

A deterministic reproduction shows:

1. A write succeeds and is readable before CommitCompact().
2. CommitCompact() returns success but stores an index offset 64 GiB too high; reading the needle returns EOF.
3. The body is still intact at its correct physical offset immediately after this commit.
4. A subsequent index-based vacuum drops the entry as unreadable and does not copy the body. The needle then returns not found.

This reproduces on 4.47. The faulty code is also present in the checked upstream master revision.

Environment and affected versions

• Linux amd64
• Large-disk build, 5BytesOffset, reported as SeaweedFS 8000GB
• Volume format v3; .vif contains "bytesOffset": 5
• .idx record layout: needle ID (8 bytes), offset (5 bytes), size (4 bytes)
• Reproduced against 4.47: c5073360007d28385a33426a42ac3e4ec504c5a3
• Source inspection confirms the same defect on master: 818f3bb71bad2fddb2aca905ee6fd05a8e8e7c3d, checked September 21, 2026 UTC

The production investigation that led to this reproduction involved 100 GiB volumes with replication 010, automatic worker vacuum on writable volumes, and an S3-backed JuiceFS workload. The reproduction requires neither JuiceFS nor S3, networking, replication, a crash, nor an actual disk error.

Root cause

Relevant source in 4.47:

https://github.com/seaweedfs/seaweedfs/blob/c5073360007d28385a33426a42ac3e4ec504c5a3/weed/storage/volume_vacuum.go#L516-L544

makeupDiff() serializes the old index entry, appends the needle to the compacted data file, then updates only bytes 8–11:

idxEntryBytes := needle_map.ToBytes(key, increIdxEntry.offset, increIdxEntry.size)

// ... append the needle to the compacted .dat at its new offset ...

util.Uint32toBytes(idxEntryBytes[8:12], uint32(offset/NeedlePaddingSize))

In the five-byte format, byte 12 holds the high offset byte. The encoding is the lower 32 offset bits in bytes 8–11, followed by the high byte at byte 12:

https://github.com/seaweedfs/seaweedfs/blob/c5073360007d28385a33426a42ac3e4ec504c5a3/weed/storage/types/offset_5bytes.go#L18-L24

Offsets are in eight-byte units, so the stale high byte contributes multiples of 2^32 * 8 = 32 GiB. The new low bits are combined with the old high bits.

For the reproduction:

Old physical offset:       68,719,476,736  (64 GiB)
Correct compacted offset:             80
Stored compacted offset:   68,719,476,816  (64 GiB + 80)
Compacted .dat size:                 232

This affects the replay of writes after the snapshot. The ordinary snapshot-copy path correctly uses ToOffset(newOffset).

The subsequent data removal follows from copyDataBasedOnIndexFile() treating EOF at the incorrect offset as an unreadable entry and skipping it:

https://github.com/seaweedfs/seaweedfs/blob/c5073360007d28385a33426a42ac3e4ec504c5a3/weed/storage/volume_vacuum.go#L703-L720

Deterministic reproduction

The following test targets 4.47. It exercises the actual compaction and commit methods in-process, using a temporary sparse file to represent a large volume with mostly obsolete data. It does not allocate 64 GiB of physical data on a sparse-file-capable filesystem and does not connect to a running cluster.

The write is deliberately inserted between CompactByIndex() and CommitCompact(). This exercises the concurrent-write replay window without timing-sensitive goroutines or sleeps.

Save the test as weed/storage/volume_vacuum_5byte_offset_test.go in a checkout of tag 4.47, then run:

go test -tags=5BytesOffset ./weed/storage \
  -run '^TestIncidentConcurrentWriteCrossesOffsetBoundary$' -count=1 -v

//go:build 5BytesOffset

package storage

import (
	"bytes"
	"os"
	"testing"

	"github.com/seaweedfs/seaweedfs/weed/storage/idx"
	"github.com/seaweedfs/seaweedfs/weed/storage/needle"
	"github.com/seaweedfs/seaweedfs/weed/storage/super_block"
	"github.com/seaweedfs/seaweedfs/weed/storage/types"
)

func TestIncidentConcurrentWriteCrossesOffsetBoundary(t *testing.T) {
	dir := t.TempDir()
	v, err := NewVolume(dir, dir, "", 784, NeedleMapInMemory, &super_block.ReplicaPlacement{}, &needle.TTL{}, 0, needle.Version3, 0, 0)
	if err != nil {
		t.Fatal(err)
	}
	defer v.Close()
	write := func(key uint64, size int) *needle.Needle {
		n := &needle.Needle{Id: types.Uint64ToNeedleId(key), Data: bytes.Repeat([]byte("x"), size)}
		n.Checksum = needle.NewCRC(n.Data)
		if _, _, _, err := v.writeNeedle2(n, true, false, false); err != nil {
			t.Fatal(err)
		}
		return n
	}
	write(1, 32)
	if err := v.DataBackend.Truncate(64 << 30); err != nil {
		t.Fatal(err)
	}
	if err := v.CompactByIndex(nil); err != nil {
		t.Fatal(err)
	}
	written := write(342511246, 113)
	before := newEmptyNeedle(uint64(written.Id))
	if _, err := v.readNeedle(before, nil, nil); err != nil || !bytes.Equal(before.Data, written.Data) {
		t.Fatalf("read before commit: %v", err)
	}
	cpd, err := os.Stat(v.FileName(".cpd"))
	if err != nil {
		t.Fatal(err)
	}
	expectedOffset := cpd.Size()
	if err := v.CommitCompact(); err != nil {
		t.Fatalf("commit: %v", err)
	}
	indexFile, err := os.Open(v.FileName(".idx"))
	if err != nil {
		t.Fatal(err)
	}
	defer indexFile.Close()
	var actualOffset int64
	if err := idx.WalkIndexFile(indexFile, 0, func(key types.NeedleId, offset types.Offset, _ types.Size) error {
		if key == written.Id {
			actualOffset = offset.ToActualOffset()
		}
		return nil
	}); err != nil {
		t.Fatal(err)
	}
	after := newEmptyNeedle(uint64(written.Id))
	_, readErr := v.readNeedle(after, nil, nil)
	t.Logf("stored offset=%d, correct offset=%d, excess=%d, post-commit read=%v", actualOffset, expectedOffset, actualOffset-expectedOffset, readErr)
	if actualOffset != expectedOffset || readErr != nil || !bytes.Equal(after.Data, written.Data) {
		physical := new(needle.Needle)
		if err := physical.ReadData(v.DataBackend, expectedOffset, written.Size, v.Version()); err != nil || !bytes.Equal(physical.Data, written.Data) {
			t.Fatalf("physical compacted body verification: %v", err)
		}
		if err := v.CompactByIndex(nil); err != nil {
			t.Fatalf("second compact: %v", err)
		}
		if err := v.CommitCompact(); err != nil {
			t.Fatalf("second commit: %v", err)
		}
		_, secondErr := v.readNeedle(newEmptyNeedle(uint64(written.Id)), nil, nil)
		_, present := v.nm.Get(written.Id)
		t.Logf("after second vacuum: read=%v, target present in index=%v", secondErr, present)
		t.Fatalf("acknowledged concurrent write lost: stored offset=%d, want=%d, read=%v", actualOffset, expectedOffset, readErr)
	}
}

newEmptyNeedle() is the existing helper in weed/storage/volume_vacuum_test.go.

Actual behavior

Output excerpts from unmodified 4.47 implementation:

volumeDataIntegrityChecking failed CheckVolumeDataIntegrity .../784.idx failed: EOF
volume 784: idx references end=68719476968 but .dat is 232 bytes; marking readonly

stored offset=68719476816, correct offset=80, excess=68719476736, post-commit read=EOF

vacuum volume 784: dropping unreadable needle key=342511246 offset=68719476816 size=118: EOF
vacuum volume 784: dropped 1 unreadable index entries (118 data bytes) during compaction

after second vacuum: read=not found, target present in index=false
--- FAIL: TestIncidentConcurrentWriteCrossesOffsetBoundary

The 118-byte needle size includes the v3 needle's metadata; the test's payload is 113 bytes.

Expected behavior

A successfully written needle must remain readable, with identical content, after compaction and commit. Replaying the write must encode all offset bytes for the selected build, including when its old and new positions lie in different 32 GiB ranges.

Compaction must not create an invalid index entry that causes a later vacuum to discard a valid body.

Suggested fix

Use the build-aware offset encoder instead of a hardcoded four-byte write:

- util.Uint32toBytes(idxEntryBytes[8:12], uint32(offset/NeedlePaddingSize))
+ OffsetToBytes(idxEntryBytes[NeedleIdSize:NeedleIdSize+OffsetSize], ToOffset(offset))

Regression coverage should include source-to-destination moves across 32 GiB and 64 GiB boundaries under 5BytesOffset, verification of both index offsets and object bodies, and survival through a second compaction. Normal four-byte builds and moves within one offset range should remain unchanged.

Production observation and confidence

The investigation started with a missing 113-byte S3 object whose write overlapped vacuum. Both registered replicas returned 404, including with readDeleted=true, while filer metadata initially still referenced the object.

Persisted job parameters show the two replicas shrinking from approximately 68.86 GiB and 55.01 GiB to approximately 19.28 GiB and 19.23 GiB. Two additional vacuums ran afterward. This fits the cross-boundary corruption and subsequent removal reproduced above.

The historical per-needle volume logs and pre-loss index offsets have rotated away, so attribution of that particular production object is high confidence rather than a complete forensic trace. The code defect and the two-compaction data-loss sequence are independently reproduced without relying on that incident.

Related fix: #11263 does not address this defect

PR #11263, included in 4.47, fixes the separate unexpected new data size failure caused by comparing a compacted snapshot against live needle-map counters.

In the same 4.47 test run:

FAIL TestIncidentConcurrentWriteCrossesOffsetBoundary
PASS TestCompactByIndex_ConcurrentWriteDoesNotFailIntegrityCheck

The snapshot-size fix does not change the four-byte offset rewrite in makeupDiff(). PR #11289, which batches fsync calls, also leaves that rewrite unchanged.