EROFS layer blobs are regularly left corrupted after an unclean shutdown: snapshots are committed without ever flushing layer.erofs to disk

Description

Nodes that lose power regularly come back with EROFS snapshots that are committed in the metadata store while their layer.erofs blob is truncated or empty. Every container that needs such a layer then fails to start, and the only recovery we have found is deleting and re-pulling the image: a plain re-pull does not rebuild an already committed snapshot.

[Fri Sep 11 08:49:26 2026] /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/61/layer.erofs: Can't lookup blockdev
[Fri Sep 11 08:49:26 2026] erofs: (device loop61): erofs_read_superblock: cannot find valid erofs superblock

failed to prepare extraction snapshot "extract-665724870-2EIl sha256:984265e5...": mount source:
"/dev/loop61", target: ".../io.containerd.snapshotter.v1.erofs/snapshots/63/fs", fstype: erofs,
flags: 1, data: "", err: invalid argument

containerd never forces the blob to stable storage before committing the snapshot: the metadata store is fsynced, the blob is not, so the snapshot survives while permanently referencing data that was never written. Same class of failure as #5854 (overlayfs), but coarser: the layer is a single filesystem image, so the whole layer dies and the mount itself fails (as in #13121).

There is no syncfs(2), fsync(2) or fdatasync(2) anywhere in the EROFS code — checked at v2.2.7 (commit 7835c8d, all links below) and on main. The blob is written and simply closed on both paths in erofsDiff.Apply() (native, tar via mkfs.erofs), and snapshotter.Commit() only stats the blob, computes disk usage and writes the metadata transaction — as does commitBlock(), which builds the blob from an overlayfs upperdir when the walking differ was used.

The documentation promises the opposite — docs/snapshotters/erofs.md sells durability as an advantage over OverlayFS in two bullets (L44-L48, L50-L52):

Parallel unpacking is now supported natively, similar to the OverlayFS snapshotter. This capability is difficult to implement in disk‑snapshot‑style snapshotters such as blockfile, devmapper and ZFS snapshotters. It also uses an efficient method to persist layer data (via fsync) compared to the OverlayFS snapshotter, which can only use syncfs;

Better data persistence guarantee: compared to directly unpacking to the host filesystem, it provides better semantics by fsyncing the individual EROFS-formatted layer blobs instead of syncfsing the whole disk each time.

We could not find that fsync in containerd or in erofs-utils on the mkfs.erofs write path — please correct me if I am looking in the wrong place. As it stands, OverlayFS with image_pull_with_sync_fs = true gives stronger guarantees than EROFS.

Steps to reproduce the issue

We have no deterministic reproducer: the race is between the kernel writeback threads and the power cut, so what follows is the setup under which we hit this sporadically, not a recipe that fails every time.

1. Run containerd with the EROFS snapshotter and differ:
   toml复制

   [plugins.'io.containerd.service.v1.diff-service']
     default = ['erofs']

2. Pull an image so its layers are unpacked by the EROFS differ.
3. Cut power to the node (or trigger a hard reset) shortly afterwards, before the kernel writeback threads flush the dirty pages.
4. Boot the node and start a container from that image: the layer fails to mount, and layer.erofs on disk is truncated or empty while the snapshot is still committed.

Describe the results you received and expected

Received: a snapshot is committed while its layer.erofs may exist only in the page cache, and after an unclean shutdown the image is permanently broken on that node.

Expected: the blob is durable before the snapshot referencing it is committed. That costs two fsync calls per layer, cheap enough to hold by default rather than behind an opt-in flag.

Proposed fix: fsync(blob) + fsync(parent dir) in Commit(), right before the metadata transaction. We are ready to prepare a PR with this change if maintainers agree with the approach and the placement.

What version of containerd are you using?

v2.2.7

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response