Proxy cache: a single transient error pushing to the local registry leaves the artifact permanently uncached

Expected behavior and actual behavior

When a client pulls through a proxy-cache project, Harbor serves the blob/manifest from upstream and asynchronously pushes it into the local registry (controller.putBlobToLocal, ManifestCache.putBlobToLocal, ManifestCache.push, ManifestListCache.push). Each of those pushes is a single attempt. If the local registry returns any error, it is logged and the push is only re-attempted when some client pulls the same content again. The original pull succeeds, so nobody notices, but until another pull happens:

• the artifact does not appear in the proxy-cache repository (the artifacts list for that repository stays empty), and
• every pull of that content is served from upstream again (a full re-transfer per pull). Content that is pulled once or rarely, e.g. a CI image pulled from a single node, never gets cached at all.

Expected: transient errors from the local registry are retried a bounded number of times before giving up.

Steps to reproduce the problem

Easiest with a multi-replica registry deployment behind a plain (non-sticky) Kubernetes Service on shared storage (e.g. an RWX NFS PVC with storage.filesystem). Core's push is a sequence of requests (POST /blobs/uploads/ → PUT …/uploads/{uuid}; PUT /manifests/{ref} → GET /manifests/{digest} read-back) that can land on different replicas via the shared HTTP connection pool. Observed on Harbor v2.13.0 with 3 registry replicas:

# core
[ERROR] [/controller/proxy/controller.go:281]: error while putting blob to local repo, http status code: 404,
  body: {"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}
[ERROR] [/controller/proxy/manifestcache.go:203]: failed to push manifest, tag: , error http status code: 404,
  body: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]}

# registry replica B, ~80 ms after replica A completed the POST / PUT
level=error msg="error resolving upload: blob upload unknown" http.request.method=PUT
level=error msg="response completed with error" err.code="manifest unknown" http.request.method=GET

After that, CheckDependencies keeps reporting the blob as missing every 20 s (Check dependency failed!) but the blob is not re-pushed, and the manifest is not written.

The same single-shot behaviour turns any other transient local error (connection reset while core streams the PUT, brief registry restart, storage hiccup) into an uncached artifact.

Versions

• harbor version: v2.13.0 (code path unchanged on main)
• docker engine / compose version: n/a (Helm deployment, registry replicas=3, storage.filesystem on an RWX PVC)

Additional context

This is complementary to #23600 / #23601, which retry the upstream fetch (mid-stream resume via Range requests). This issue is about the local side: the push into Harbor's own registry.

Proposed fix (PR to follow): wrap local.PushBlob (re-opening the upstream reader per attempt) and local.PushManifest in a small bounded retry (3 attempts, 200 ms → 400 ms backoff, no retry on context cancellation). With the retry, the manifest read-back case is fully covered (the first PUT already landed, so the read-back succeeds on any replica after the backoff), and the blob-upload case succeeds whenever a later attempt keeps POST and PUT on the same replica.