Implement Docker Hub Proxy for Docker Image Caching

Feature description

In a CI/CD process like Kestra, it's common to encounter Docker Hub rate limits when pulling Docker images repeatedly during builds. To mitigate this issue and improve build performance, Docker Hub proxy feature within Kestra could improve performance and remove this rate limiting.

Problem Statement

Currently, CI/CD workflows relying on Docker images from Docker Hub (or any other) can face rate limits, causing delays and disruptions in build processes. This limitation hinders the scalability and efficiency of automated builds, especially in large-scale projects with frequent image pulls.

Ex for anonymous user : 100 pulls per 6 hours per IP address

Proposed Solution

Integrate a Docker Hub proxy mechanism within Kestra that allows caching Docker images locally. This proxy should intelligently manage image requests, reducing the need for repetitive pulls from Docker Hub and optimizing build times.

Benefits

1. Improved Build Performance: By caching Docker images locally, CI/CD builds will experience faster execution times as they won't be dependent on external Docker Hub requests for every build.
2. Reduced Dependency on External Sources: Minimizing direct dependencies on Docker Hub reduces the impact of rate limits and network latency, enhancing the reliability of CI/CD pipelines.
3. Scalability: The Docker Hub proxy feature enables seamless scaling of CI/CD infrastructure without concerns about exceeding Docker Hub rate limits.
4. Enhanced Developer Experience: Developers can focus on coding and testing without being hindered by external service limitations, leading to a smoother development workflow.

Implementation Considerations

1. Proxy Configuration: Provide a straightforward configuration option within Kestra's settings to define the Docker Hub proxy.
2. Cache Management: Implement intelligent caching strategies to manage cached Docker images efficiently, considering expiration policies and cache invalidation mechanisms.
3. Logging and Monitoring: Include logging and monitoring capabilities to track proxy activity, cache hits/misses, and overall performance metrics.
4. Documentation: Ensure comprehensive documentation detailing how users can configure and utilize the Docker Hub proxy feature in their CI/CD workflows.

Use Case

Consider a scenario where a CI/CD pipeline in Kestra regularly pulls Docker images from Docker Hub for building and testing applications. With the Docker Hub proxy feature enabled, these images are cached locally, significantly reducing build times and improving overall pipeline efficiency.

This feature request aims to enhance the functionality and performance of Kestra in CI/CD environments, providing users with a seamless Docker image caching solution to optimize build workflows.

Relevant Documentation

1. Docker setup Proxy
2. Gitlab to setup proxy (in a gitlab CI/CD)
3. harbor proxy setup