garbage-collect --delete-untagged sometimes removes a blob that is still referenced by multiple live, tagged manifests

Description

Running garbage-collect --delete-untagged against a registry with only single-arch (application/vnd.docker.distribution.manifest.v2+json) manifests occasionally deletes a blob that is still referenced by multiple live, tagged manifests. This has happened repeatedly over several weeks, hitting different shared base layers each time, and has corrupted a number of unrelated images that all happened to share a base layer with whatever got swept.

This is not a multi-arch / manifest-list issue, none of the affected repositories use manifest lists, and I've confirmed the manifest mediaType on every broken tag is plain manifest.v2+json.

Reproduce

Unfortunately I don't have a minimal reproduction, this appears to be non-deterministic and has only shown up over repeated nightly runs against a registry with a few hundred repositories and several years of accumulated tags/base image reuse. I'm not able to reliably trigger it on demand.

Expected behavior

A blob referenced by a live, tagged manifest should never be marked eligible for deletion, and this should hold true consistently across repeated runs against unchanged tag state.

registry version

$ registry --version registry github.com/distribution/distribution/v3 3.1.1

Additional Info

On one specific nightly run, a blob (sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593) that is referenced by 9 different repositories across latest-master and next-master tags was deleted by GC. None of those tags had changed — no push, no retag, no deletion — before or after that run.

The interesting part is that the exact same blob had been correctly identified as reachable ("marking blob") in at least two prior nightly runs against the same tag state:

docker_raptor1: marking blob sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593

(seen in two separate runs before the incident)

Then, on the run that deleted it, none of the 9 referencing repositories produced a "marking blob" line for this digest at all — it went straight to:

time="2026-08-21T00:20:34.057620072Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/a8/a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593"

followed by the layer link deletions for all 9 repos:

docker_raptor3: layer link eligible for deletion: sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593
docker_raptorg3000: layer link eligible for deletion: sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593
docker_raptormx: layer link eligible for deletion: sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593
docker_raptor1: layer link eligible for deletion: sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593
docker_raptor2: layer link eligible for deletion: sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593
docker_raptorftd: layer link eligible for deletion: sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593
docker_raptor4: layer link eligible for deletion: sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593
docker_raptor1_norcim: layer link eligible for deletion: sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593
docker_raptorios: layer link eligible for deletion: sha256:a863d105875ced3e22129c07eda3fd79df7e64799a1ebbc0b737b531b2091593

Tag state was unchanged across all of these runs — nothing was pushed, retagged, or deleted between the "correctly marked" runs and the run that deleted it. The only difference is which run it happened to be.

We recovered the blob from a backup taken a few hours before the deletion, restored it into the live blob store, and confirmed via sha256sum that it matches the digest exactly. All 9 previously-broken tags are now pulling correctly again, which further confirms this wasn't a legitimate deletion — the blob was genuinely still in use.

This isn't an isolated incident. Over about two weeks of running --delete-untagged nightly, we've had this happen to several different, unrelated shared base layers, each time silently corrupting every image that depended on that layer, while the manifests themselves remained intact and tagged. We only noticed because we built a script that walks every manifest and checks whether its referenced blobs actually exist on disk after each run.