Proxy cache serves 404 when the Docker Hub login endpoint is rate limited
Expected behavior Pulling through a Docker Hub proxy-cache project authenticated with a Docker Hub account on the Team tier, without any pull rate limits, succeeds and does not produce a rate-limit error.
Actual behavior
When multiple not-yet-cached images are requested (when renovate bumps our dependencies in the git repo, and CI tries to build it), harbor-core logs produce
[ERROR] [/server/middleware/repoproxy/proxy.go:162]: failed to proxy manifest, fallback to local,
request uri: /v2/cache/library/node/manifests/sha256:d32cdf619f63fe0471182d08996dd516c6275bb5fd31ae06e55a570bd9e1ad43,
error: login to dockerhub error: login to dockerhub error: {"detail": "Rate limit exceeded", "error": false}and clients receive 404 not found for images that exist upstream but are not yet in the proxy cache.
It seems to me that Harbor is calling Docker Hub's /login endpoint on every pull instead of caching credentials and refreshing them only when their TTL is nearing expiration.
Steps to reproduce the problem:
- Add a Docker Hub registry endpoint with credentials (password or PAT).
- Create a proxy-cache project using that endpoint.
- Concurrently pull a number of not-yet-cached multi-arch tags through the project — a few dozen parallel pulls, as a CI burst produces.
harbor-corelogslogin to dockerhub error: {"detail": "Rate limit exceeded", "error": false}, and clients get 404 for the uncached tags. Cached tags keep working.
Versions: Please specify the versions of following systems.
- harbor version: 2.15.1
- docker engine version: 29.3.0 / containerd via GitLab Kubernetes executor
- helm chart version: 1.19.1
Additional context:
This issue began appearing after I upgraded Harbor from version 2.13.0 to 2.15.1.
Source: goharbor/harbor