repo verify --drop cannot recover pebbleds repo when datastore iteration fails on a corrupt SST

Checklist

• This is a bug report, not a question. Ask questions on discuss.ipfs.tech.
• I have searched on the issue tracker for my bug.
• I am running the latest kubo version or have an issue updating.

Installation method

built from source

Version

Affected repository was created and the problematic SST was written under:

Kubo version: 0.41.0-d719fb87b
Repo version: 18
System version: amd64/linux
Golang version: go1.26.2

The same affected repository remained unreadable/unrecoverable after upgrading to:

Kubo version: 0.43.0-e9914bb47
Repo version: 18
System version: amd64/linux
Golang version: go1.26.5

Config

Relevant information without the sensitive information I could have in config like ips:
{
  "Datastore": {
    "BlockKeyCacheSize": null,
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "path": "pebbleds",
      "type": "pebbleds"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "800GB"
  }
}

Description

Description

I encountered an SST corruption while using Kubo with the experimental pebbleds datastore.

The underlying cause of the SST corruption is being investigated separately and may be in Pebble or below the database layer. This issue is specifically about Kubo's recovery behavior once a datastore iteration error occurs.

A single unreadable SST caused datastore iteration to fail, and as a result ipfs repo verify --drop was unable to continue far enough to identify/drop the affected blocks.

My understanding from discussion with the IPFS team is that verification enumerates the datastore through AllKeysChan. Once the datastore returns an iteration error, the sweep terminates, so repo verify --drop cannot reach the affected entries.

I understand that supporting recovery from arbitrary corruption across multiple datastore implementations may be difficult. I'm opening this issue primarily to document this failure mode and discuss whether Kubo can provide a safer or clearer recovery path.

Kubo versions

The affected SST was written while the node was running Kubo 0.41.0.

The node was later upgraded to Kubo 0.43.0. The affected repository remained unreadable/unrecoverable after the upgrade, so upgrading did not resolve the existing datastore problem.

To be clear, I am not suggesting that both versions independently caused corruption. The affected repository originated under 0.41.0 and remained affected under 0.43.0.

Datastore configuration

The relevant datastore configuration from the preserved affected repository was:

{
  "Datastore": {
    "BlockKeyCacheSize": null,
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "path": "pebbleds",
      "type": "pebbleds"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "800GB"
  }
}

disableWAL was not explicitly configured.

The affected repository was approximately 201 GB.

The complete affected repository has been preserved, and the production node was subsequently rebuilt with a fresh repository.

Initial symptom

I ran:

ipfs repo gc

GC successfully removed a large number of blocks before eventually terminating with:

Error: unexpected EOF

A subsequent attempt to run GC returned:

Error: EOF

The repository could then no longer be used normally.

Verification

Running repository verification exposed Pebble on-disk corruption involving:

pebbleds/67232266.sst

with a CRC32C checksum mismatch involving keys under:

/blocks/...

However, running:

ipfs repo verify --drop

could not recover the repository.

My understanding is that repo verify enumerates the datastore using AllKeysChan. When Pebble encounters the problematic data and returns an iteration error, the entire sweep terminates. Therefore, --drop cannot continue far enough to individually identify/drop the affected block(s).

Additional Pebble diagnostics

I preserved the original repository and ran the Pebble CLI matching the Pebble version used by Kubo:

github.com/cockroachdb/pebble/v2/cmd/[email protected]

Running:

pebble db check <kubo-repo>/pebbleds

completed with:

Found 1 WALs

  - 67498829: {(<kubo-repo>/pebbleds,000)}

[JOB 1] WAL 67498829: {(<kubo-repo>/pebbleds,000)} stopped reading at offset:
(<kubo-repo>/pebbleds/67498829.log: 0); replayed 0 keys in 0 batches

checked 35462601 points and 0 tombstone

No additional corruption was reported while checking approximately 35.4 million points.

The affected SST is:

67232266.sst

and its physical size is:

33744727 bytes

The Pebble manifest also records exactly:

size:33744727

and places the table at L6.

The SST filesystem timestamp is:

2026-08-08 00:29:50.195105098 -0400

while the manifest records:

2026-08-08T04:29:50Z

which represents the same moment.

A standalone:

pebble sstable check <kubo-repo>/pebbleds/67232266.sst

currently completes without reproducing the checksum error.

The underlying corruption/checksum behavior is being investigated separately as a Pebble/storage issue.

System checks

I inspected the system and kernel journals around the time the SST was written.

I did not find evidence in that window of:

• OOM kill
• host reboot/shutdown
• NVMe error
• I/O error
• ext4 error
• RAID failure
• kernel panic
• Kubo service restart

The host uses software RAID1 and currently reports both devices healthy ([UU]) with mismatch_cnt of 0.

The machine does not have ECC RAM.

Expected behavior / feature request

I understand that reliably recovering from arbitrary datastore corruption may be difficult, especially across different datastore implementations.

However, it would be useful if ipfs repo verify --drop could handle a datastore iteration failure more gracefully when using an LSM datastore such as pebbleds.

Ideally, one unreadable SST/range should not prevent verification of the entire remaining repository.

Potential approaches could include:

• preserving partial iteration progress
• surfacing the affected key range/file where possible
• allowing verification to skip an unreadable range and continue
• providing a pebbleds-specific recovery mechanism
• or, at minimum, producing a clearer message explaining that --drop cannot repair this class of pebbleds corruption

I'm not assuming any of these approaches are straightforward to implement. The main goal of this issue is to document the recovery limitation and discuss whether a better recovery path is possible.

Additional context

We use pebbleds because these Kubo nodes handle large, continuously growing datasets with a very large number of IPFS blocks. Pebble was selected as a better fit for this type of large datastore/workload than FlatFS.

I understand that pebbleds is still experimental in Kubo and that FlatFS provides different failure characteristics, where corruption of an individual block file is more isolated.

I still have the complete ~201 GB affected repository preserved and can run additional read-only diagnostics or tests if they would be useful.