firstMatching silently skips a record that fails its checksum

Observed behavior

An ordered consumer silently skips a file-store record with an invalid checksum and successfully delivers the next record, without reporting an error or sequence gap.

On a standalone server, I created stream T with subjects t.>, file storage, and no per-subject limit. Two publishes were acknowledged:

• A on t.a, sequence 1
• B on t.b, sequence 2

After stopping the server, I flipped one bit in sequence 1's trailing 8-byte record checksum. This changes no length, sequence, timestamp, or subject bytes. After restarting, an ordered consumer with filter t.> and DeliverAll delivered only sequence 2. No error or gap was reported, and the ordered consumer's gap detection did not trigger.

Output:

A=seq 1  B=seq 2
delivered seq 2 subject t.b data "B"
delivered=[2]

The same enumeration on an undamaged stream delivers both records.

Source inspection points to the following path in server/filestore.go:

• indexCacheBuf performs record-length sanity checks, deferring the per-message checksum validation until the record is read.
• msgFromBufEx returns errBadMsg{"invalid checksum"} for the damaged record.
• msgBlock.firstMatching returns early for errPartialCache and errNoCache, but continues past other cacheLookup errors, including the checksum failure.
• msgBlock.firstMatchingMulti also discards errors at its three cacheLookup sites. Two continue on any error; the third ignores the failed lookup and returns true.

These functions were identical in v2.15.0 and the main branch checked on 2026-09-21.

Skipping an unreadable record may be an intentional resilience choice. The problem is that the enumeration succeeds without making the skipped record visible to the caller.

Expected behavior

An unreadable record should not silently disappear from a successful enumeration.

The consumer should receive an explicit error or a reported sequence gap, or both records should be delivered if recovery is possible. If skipping is intentional, there should be an observable indication of the skipped record.

Server and client version

Server: v2.15.0 Client: nats.go V1.53.1

Host environment

Linux/arm64, server embedded in a go test process. File storage in a temporary directory created by t.TempDir(). Standalone, single server. No clustering.

Steps to reproduce

repro_test.zip

Use repro_test.go (in zip above) with this go.mod:

module repro

go 1.26.5

require (
   github.com/nats-io/nats-server/v2 v2.15.0
   github.com/nats-io/nats.go v1.53.1
)

Run: go test -v -count=1 ./...

The reproduction performs these steps:

1. Start an embedded server.
2. Create stream T with subjects t.> and file storage, leaving other settings at their defaults
3. Publish A to t.a and B to t.b, receiving acknowledgments for sequences 1 and 2
4. Stop the server
5. Walk <StoreDir>/jetstream/$G/streams/T/msgs/*.blk, locate the record for sequence 1, and XOR its last byte with 0x01. This changes only one bit of its trailing checksum.
6. Restart the server using the same store directory
7. Create the subscription:

js.SubscribeSync("t.>", nats.OrderedConsumer(), nats.DeliverAll())

1. Collect deliveries and check for errors or a reported gap. The defect test fails by design when the issue reproduces. The undamaged control passes:

  --- FAIL: TestEnumerationSilentlySkipsUnreadableRecord
      seq 1 was never delivered and no error or gap was reported ... the walk returned seq 2 successfully
  --- PASS: TestControlUndamagedDeliversBoth

Possibly related

#7549 reports acknowledged writes lost after low-level .blk damage on a replicated stream. The symptoms may be related, but this reproduction does not establish a shared cause.

Submission acknowledgment

• I am a human being writing in my own words and not an AI agent. I will not use an AI agent to communicate on my behalf in this issue, either directly or via copy-paste.