local cache exporter: reset=true can delete blobs from a concurrent export
Description
reset=true on the local cache exporter can delete blobs that another
concurrent export just registered, leaving index.json with a tag whose
manifest blob no longer exists. That tag then fails to import permanently.
resetCacheStore reads index.json and then walks and deletes unreferenced
blobs. ociindex.Read() takes a flock, but the subsequent cs.Walk / cs.Delete
runs outside it, so a concurrent export that adds a manifest between the read and
the delete has its blob collected.
Reproduce
Two Dockerfiles with different content:
$ printf 'FROM busybox\nRUN echo A > /a1\nRUN echo A2 > /a2\n' > Dockerfile.a
$ printf 'FROM busybox\nRUN echo B > /b1\nRUN echo B2 > /b2\n' > Dockerfile.bExport both to the same directory concurrently:
$ docker buildx build -f Dockerfile.a \
--cache-to type=local,dest=./cd,tag=a,mode=max,reset=true \
--output type=cacheonly . &
$ docker buildx build -f Dockerfile.b \
--cache-to type=local,dest=./cd,tag=b,mode=max,reset=true \
--output type=cacheonly . &
$ waitThen check index.json against the blob files:
$ python3 -c "
import json,os
d=json.load(open('cd/index.json'))
for m in d['manifests']:
t=(m.get('annotations') or {}).get('org.opencontainers.image.ref.name')
f='cd/blobs/sha256/'+m['digest'].split(':')[1]
print(t, m['digest'][:20], os.path.exists(f))
"
a sha256:6ad01113a4a1f8f True
b sha256:fb1a0f3d2c9e7b2 FalseImporting the affected tag fails from then on:
WARNING: local cache import at ./cd skipped: digest sha256:fb1a0f3d... unavailable:
content sha256:fb1a0f3d...: not foundReproduced in 2 of 3 runs. docker buildx bake with two targets sharing one
dest hits the same path.
Control
The same concurrent export without reset=true never produced a dangling
reference in my runs. It loses one tag's index.json entry instead (last writer
wins), which is recoverable by re-exporting. With reset=true the directory is
left inconsistent and the affected tag cannot be recovered without a full
re-export.
Expected
Either serialize the reset against concurrent writers, or scope collection to blobs that were unreferenced at the time the lock was taken.
Version
docker buildx version: v0.35.0-desktop.2
BuildKit (daemon): v0.32.2
Docker Engine: 29.6.2Source: moby/buildkit