proxyBlobStore.Delete is an unconditional stub, so OnBlobExpire always fails and blob-level TTL never reclaims proxy cache storage
Description
When running the registry as a pull-through cache (proxy.remoteurl configured) with proxy.ttl set, blob-level TTL expiration never actually removes blob content from storage. Every blob expiry logs an error and the blob is left permanently on disk, still linked from its repository, so a later registry garbage-collect also refuses to remove it (it's still referenced).
Root cause
OnBlobExpire in registry/proxy/proxyregistry.go does:
s.OnBlobExpire(func(ref reference.Reference) error {
...
repo, err := registry.Repository(ctx, r)
...
blobs := repo.Blobs(ctx)
// Clear the repository reference and descriptor caches
err = blobs.Delete(ctx, r.Digest())
if err != nil {
return err // <-- returns before v.RemoveBlob() below
}
err = v.RemoveBlob(r.Digest().String())
if err != nil {
return err
}
return nil
})repo.Blobs(ctx) for a proxied repository returns a proxyBlobStore, whose Delete is an unconditional stub in registry/proxy/proxyblobstore.go:
func (pbs *proxyBlobStore) Delete(ctx context.Context, dgst digest.Digest) error {
return distribution.ErrUnsupported
}So blobs.Delete always returns ErrUnsupported, OnBlobExpire always returns early, and v.RemoveBlob (the call that actually vacuums the blob from storage) is never reached. Confirmed still present on main as of this writing, and v3.1.1 is the latest tagged release, so there is no currently-released version where this works.
Observed behavior
With log.level: warn or higher, every blob-level expiry logs, e.g.:
level=error msg="Scheduler error returned from OnExpire(myorg/myimage@sha256:<digest>): operation unsupported"The blob's _layers/sha256/<digest> link under the repository is never removed (since the Delete call that would remove it never succeeds), so the blob remains "referenced" from garbage-collect's point of view and is never swept either. OnManifestExpire has no equivalent stub and works correctly, so tag/manifest references do get cleaned up on schedule — only the underlying blob content is stuck.
Net effect: for any pull-through cache with proxy.ttl configured, blob storage grows unbounded over time. Neither the TTL scheduler nor registry garbage-collect will ever reclaim it; the only way to reclaim space is to wipe the storage backend directly.
Expected behavior
Blob-level TTL expiry on a proxy/pull-through repository should actually free the blob (either by giving proxyBlobStore.Delete a real implementation that unlinks the repo-local reference, or by having OnBlobExpire tolerate/ignore ErrUnsupported from that call and proceed to v.RemoveBlob regardless).
Version
registry github.com/distribution/distribution/v3 3.1.1 (also reproduced by inspection against main)
Reproduction
- Configure the registry as a pull-through cache with a short
proxy.ttl(e.g.10m). - Pull any image through it.
- Wait past the TTL and check logs for
Scheduler error returned from OnExpire(...): operation unsupported. - Confirm the blob file is still present under
/blobs/sha256/...and still linked under the repository's_layers/sha256/.... - Run
registry garbage-collect --dry-run— the blob is still marked as referenced and reported as not eligible for deletion.
Source: distribution/distribution