`docker build` fails with "ref moby/1/<id> locked for <d>: unavailable" — overlay differ fallback reuses the ingest ref, and the overlay view mount is missing `index=off`

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Environment

• Docker Engine 29.x with the containerd image store (the default since 29)
• containerd 2.x, overlayfs snapshotter
• rootless (dockerd-rootless / DinD), kernel 5.4 (TLinux) (kernels >= 5.8 are not affected by defect A — see below)
• BuildKit v0.31.x
• cat /sys/module/overlay/parameters/index → Y

Symptom

During the export layers phase of docker build (single-platform, ordinary multi-layer Dockerfile):

#16 ERROR: mount callback failed on /home/rootless/.local/share/docker/containerd/daemon/tmpmounts/containerd-mount1808443785: mount callback failed on /home/rootless/.local/share/docker/containerd/daemon/tmpmounts/containerd-mount541557879: failed to open writer: ref moby/1/j8k1rgarc9wh9vckf6bo73ec3 locked for 75.870687ms (since 2026-09-17 06:40:56.639537157 +0000 UTC m=+2455.066062373): unavailable

Related: moby/moby#52431, moby/moby#52607, moby/buildkit#3270.

Root cause: two independent defects chained together

Defect A — the overlay differ's read-only view mount omits index=off

overlay.WriteUpperdir mounts a read-only overlay "view" of the upperdir using nothing but lowerdir=<snapshot>/fs:<empty temp dir>, so the kernel default applies (index=on on most distros). But containerd's own overlayfs snapshotter always mounts these same directories with index=off (and userxattr when needed) — see plugins/snapshots/overlay/overlay.go.

As a result the view mount fails with EBUSY whenever the directory is in use as the upperdir/workdir of another overlay mount. The kernel says so itself:

# directory already in use as upperdir of another overlay mount
$ mount -t overlay overlay -o lowerdir=/tmp/ovl/upper:/tmp/empty1 /tmp/v1
mount: /tmp/v1: overlay already mounted on /root/t/mnt.
$ echo $?
32

$ mount -t overlay overlay -o lowerdir=/tmp/ovl/upper:/tmp/empty2,index=off /tmp/v2
$ echo $?
0

$ dmesg -T | tail -2
overlayfs: lowerdir is in-use as upperdir/workdir of another mount, mount with '-o index=off' to override exclusive upperdir protection.
overlayfs: lowerdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.

Production log (the failing mount, note there is no index=off):

failed to compute blob by overlay differ (ok=false): failed to write compressed diff:
mount callback failed on /run/user/1000/containerd-mount520831114: failed to mount
/run/user/1000/containerd-mount3849756922: mount source: "overlay", fstype: overlay,
data: "lowerdir=/home/rootless/.local/share/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/47/fs:/home/rootless/.local/share/docker/tmp/buildkit440776761",
err: device or resource busy

Corroboration: setting the kernel parameter overlay.index=off (/sys/module/overlay/parameters/index=N) on the affected hosts makes the failures disappear. That is a global change though; the fix should be per-mount (Defect A's patch).

Kernel version dependency. Since commit f0e1266ed24b ("ovl: fix mount option checks for nfs_export with no upperdir", Amir Goldstein) a non-upper (lowerdir-only) mount forces index = false:

	/* Workdir/index are useless in non-upper mount */
	if (!config->upperdir) {
		...
		if (config->index && index_opt) {
			pr_info("option \"index=on\" is useless in a non-upper mount, ignore\n");
			index_opt = false;
		}
		config->index = false;
	}

so on kernels that contain that commit the view mount is not rejected and Defect A does not bite. That commit was first released in v5.8-rc6, so kernels >= 5.8 are not affected by Defect A. Our hosts run 5.4 (TLinux) and are affected. This explains why the failure is rarely reported: most kernels in use today hide Defect A.

Note this makes Defect B the one that actually matters: it is what turns any failure of the overlay differ into a hard build failure, on every kernel. Defect A is still worth fixing — it removes the trigger on older kernels and aligns BuildKit with what containerd does for the same directories — but it is not sufficient on its own.

Defect B — the fallback reuses the same ingest ref, which is guaranteed to collide

When the overlay differ fails after it has opened the content writer, computeBlobChain falls back to the containerd differ with the same reference:

desc, err = sr.cm.Differ.Compare(ctx, lower, upper,
    diff.WithMediaType(mediaType),
    diff.WithReference(sr.ID()),      // ← same ref the overlay differ just used
    diff.WithCompressor(compressorFunc),
)

But containerd's ingest lock is not released synchronously by Close():

• the gRPC proxy's Close() is a fire-and-forget CloseSend() (core/content/proxy/content_writer.go)
• the local writer unlocks only after fp.Sync() (plugins/content/local/writer.go)
• and the ingest key (bref = <ns>/<seq>/<ref>) is deterministic for a given ref, so the new writer lands on exactly the same lock

so the fallback always races with the lock still being released. Measured in production: the collision was reported ~3 ms after the overlay failure, i.e. as soon as the fallback issued its Writer() — the window only has to exceed a couple of gRPC round trips, so this is not a rare timing accident.

Minimal deterministic reproducer, no BuildKit involved (~10 s):

w, _  := cs.Writer(ctx, content.WithRef(ref))   // namespace "moby"
io.Copy(w, bytes.NewReader(make([]byte, 256<<20)))
w.Close()                                        // CloseSend(): does not wait
_, err := cs.Writer(ctx, content.WithRef(ref))
// => ref moby/1/<ref> locked for <d>: unavailable

Full program: contentlock/main.go (attached, go run ./contentlock).

Defect C (bonus) — Abort runs before Close, so it always fails

cache/blobs_linux.go aborts the ingest while the writer is still open:

failed to abort writer "sfen5uoonojh1ze6zbtmakb1s" error="unlinkat
/home/rootless/.local/share/docker/containerd/daemon/io.containerd.content.v1.content/ingest/8d112d5d69908e8d1bd85e0fd091a89a2bf23c20d0c7227fece20a530f06c270:
directory not empty"

which rolls back the metadata transaction that removes the ingest bucket and leaks the ingest directory on disk (possibly the same leak as moby/moby#46136).

Workarounds available today

1. echo N > /sys/module/overlay/parameters/index kernel parameter — verified to stop the failures on our hosts.
2. "features": {"containerd-snapshotter": false} — already suggested in moby#52607; not viable for multi-platform builds.
3. Serialise concurrent builds + retry the build.

Proposed fix

Two independent patches:

• 0001 — add index=off to the overlay view mount, matching what containerd does for the same directories. Guarded by the same kernel capability check containerd uses, so the option is not sent to kernels that do not know it.
• 0002 — use a unique ingest ref per differ attempt (keeping sr.ID() as a prefix so ingests stay traceable), abort the ingest when a fallback differ fails, and close the writer before aborting it.

0002's two changes belong together: closing before aborting is only safe once references are unique.

Not yet identified

Which mount holds snapshots/<N>/fs as its upperdir/workdir at the moment the view mount is attempted.

Defects A, B and C are reproducible and fixable regardless of this open question.