containerd image store: `docker push` aborts a large layer with "net/http: timeout awaiting response headers" when the registry needs >30s to finalize the blob; classic overlay2 path succeeds on the same registry

Description

On Docker Engine 29.x with the containerd image store (the default for fresh installations), docker push aborts a multi-gigabyte layer with:

failed commit on ref "layer-sha256:<digest>": failed to do request:
Put "https://<registry>/v2/<repo>/blobs/uploads/<uuid>.patch?digest=sha256%3A<digest>":
net/http: timeout awaiting response headers

The same image, pushed to the same registry from the same network, succeeds when the daemon is configured with "features": { "containerd-snapshotter": false } (classic overlay2 graph driver). In that configuration the client waits for the registry and the push completes.

The trigger is a registry that takes longer than 30 seconds to return response headers for the final blob commit request. The byte upload itself completes quickly; the registry then spends minutes checksumming and finalizing the blob server-side before responding. We observed several minutes for a single multi-GB layer against a commercial artifact-repository product. The classic path tolerates this; the containerd path does not.

Expected behaviour

Pushing an image through the containerd image store should be at least as tolerant of a slow registry as the classic graph-driver path — either by not imposing a response-header timeout on the blob-commit request, by retrying on that timeout, or by exposing the timeout as a daemon configuration option.

Actual behaviour

The push aborts on the first occurrence, with no retry and no backoff.

The cause appears to be a hardcoded 30-second ResponseHeaderTimeout on the HTTP transport used by the containerd image store's registry client:

• containerd/containerd, core/remotes/docker/registry.go, DefaultHTTPTransport():

  func DefaultHTTPTransport(defaultTLSConfig *tls.Config) *http.Transport {
      return &http.Transport{
          // ...
          TLSHandshakeTimeout:   10 * time.Second,
          ExpectContinueTimeout: 5 * time.Second,
          ResponseHeaderTimeout: 30 * time.Second,   // <-- applies to the blob commit PUT
      }
  }
  

• moby/moby, daemon/hosts.go, (*Daemon).RegistryHosts() reaches that transport via hostconfig.ConfigureHosts(...), and daemon/containerd/resolver.go (newResolverFromAuthConfig) passes the resulting RegistryHosts straight into docker.NewResolver.

Thirty seconds is the entire budget the registry is given to checksum and finalize a blob that may be several gigabytes. We are not aware of any daemon.json key that exposes this value.

The classic path has no equivalent limit. moby/moby, daemon/pkg/registry/registry.go, newTransport() sets TLSHandshakeTimeout and IdleConnTimeout but deliberately sets no ResponseHeaderTimeout, so it waits for the registry for as long as the connection stays alive. That single difference accounts for the whole behavioural gap between the two image stores.

Two further details confirm which code path is in play:

1. The error string failed commit on ref "layer-sha256:..." originates from containerd's pusher (core/remotes/docker), not from the classic docker/distribution client.
2. The classic path emits Retrying in N seconds lines before failing. Those lines are absent here — the containerd path gives up immediately.

One caveat on our own measurements, in case it helps anyone triaging a similar report: in our production failure the abort was logged about five minutes after the previous Pushed line, which initially led us to suspect a five-minute timeout. With five concurrent uploads in flight, that interval is simply when the large blob's data transfer finished and its commit PUT was issued; the abort follows 30 seconds later. The 30-second constant is consistent with everything we observed.

Impact

This is not an exotic configuration. Because the containerd image store is enabled by default on fresh installations of Docker Engine 29.0 and later, while hosts upgraded in place keep overlay2, a fleet of build agents silently diverges: newly provisioned agents fail a push that older agents complete successfully, with an identical Dockerfile, identical engine version, and an identical --no-cache --pull build. In our case two agents differed in no other relevant respect (same engine 29.7.2, same containerd, same runc, same DNS, same route, comparable TLS/connect timings to the registry), and the storage backend was the only variable.

The only workaround we found is to disable the containerd image store — that is, to opt out of the new default. That is the outcome we would like to avoid; see the questions at the end.

---

Steps to reproduce

This reproduction is fully self-contained. It needs only a throwaway VM, Docker Engine 29.x, the public registry:2 image, and mitmproxy. No commercial registry is required — the slow blob commit is simulated deterministically.

1. Build a large, incompressible single-layer image

# Dockerfile.bigblob
FROM ubuntu:24.04
RUN dd if=/dev/urandom of=/big.bin bs=1M count=6000

/dev/urandom is required: a file of zeros compresses to almost nothing and the resulting blob would never be large enough to matter.

2. Start a local registry

docker run -d -p 5000:5000 --name registry --restart=always registry:2

3. Put a proxy in front of it that delays only the blob-commit request

# slow_commit.py
# Run with:
#   mitmdump -s slow_commit.py --mode reverse:http://127.0.0.1:5000 -p 5001
import time
from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    # Delay only the final blob commit: PUT .../blobs/uploads/<uuid>?digest=sha256:...
    if flow.request.method == "PUT" and "digest" in flow.request.query:
        time.sleep(60)   # anything above the hardcoded 30s ResponseHeaderTimeout

Delaying only the commit is deliberate. It leaves the byte upload running at full speed and isolates exactly the behaviour a slow registry exhibits: fast transfer, slow finalization.

4. Run A — containerd image store (fails)

sudo tee /etc/docker/daemon.json >/dev/null <<'EOF'
{
  "features": { "containerd-snapshotter": true },
  "insecure-registries": ["127.0.0.1:5001"]
}
EOF
sudo systemctl restart docker

docker info -f 'driver={{.Driver}} status={{.DriverStatus}}'
# expected: driver=overlayfs status=[[driver-type io.containerd.snapshotter.v1]]

docker build -f Dockerfile.bigblob -t 127.0.0.1:5001/test/bigblob:1 .
time docker push 127.0.0.1:5001/test/bigblob:1

Result: aborts 30 seconds after the commit PUT is issued, with failed commit on ref "layer-sha256:..." : ... net/http: timeout awaiting response headers. No retry lines are emitted.

5. Run B — classic overlay2 (succeeds)

Only the feature flag changes. Same image, same registry, same delay.

sudo tee /etc/docker/daemon.json >/dev/null <<'EOF'
{
  "features": { "containerd-snapshotter": false },
  "insecure-registries": ["127.0.0.1:5001"]
}
EOF
sudo systemctl restart docker

docker info -f 'driver={{.Driver}}'
# expected: driver=overlay2

docker build -f Dockerfile.bigblob -t 127.0.0.1:5001/test/bigblob:1 .
time docker push 127.0.0.1:5001/test/bigblob:1

Result: the push waits out the full 60-second delay and completes successfully.

sudo journalctl -u docker -f during both runs shows the underlying HTTP exchange and the presence or absence of retries.

Notes on the reproduction

The layer size is not the trigger and can be reduced substantially — a large layer is only what makes a real registry slow enough to cross the threshold in practice. The trigger is purely time.sleep(n) with n > 30. Conversely, setting n < 30 makes both runs pass, which is a useful negative control.

---

Related upstream work

containerd's resolver has since gained doWithTransportRetries in core/remotes/docker/resolver.go, which retries transient transport errors — explicitly including response-header timeouts — up to maxAttempts = 5. If we are reading this correctly, an engine built against a containerd containing that change should no longer exhibit the failure. Confirmation of that, and of which engine release will carry it, would be very helpful.

---

Questions

The reason for opening this issue is less about the workaround (which we have) and more about how to stay on the new default:

1. Is the 30-second ResponseHeaderTimeout intentional for the blob-commit request, and can it be made configurable? Thirty seconds is a reasonable default for a manifest HEAD or a token request, but the commit PUT is a request whose entire purpose is to wait for server-side finalization of a potentially multi-gigabyte blob. Either exempting that request from the timeout, or exposing the value through daemon.json, would resolve this class of failure. Today there appears to be no supported way to adjust it without recompiling.

2. Can the containerd push path adopt the same retry/backoff semantics as the classic path? The asymmetry between the two backends is the surprising part: identical engine, identical registry, different resilience, with nothing in docker info suggesting that pushing behaviour would differ.

3. What is the recommended configuration for a fresh Docker Engine 29.x installation that must push very large layers to a slow-finalizing registry, while keeping the containerd image store? We would prefer not to set "features": { "containerd-snapshotter": false } on every newly provisioned host, because that means permanently opting out of the default and out of the direction the project is moving in. Is there a supported way to influence the registry client used by the containerd image store from dockerd — for example per-registry host configuration, hosts.toml-style settings, or any timeout tuning — that we have missed?

4. Is any of this documented? The behavioural difference between the two image stores on push is not mentioned on the containerd image store page, which currently describes the switch as transparent for most users. A note about large-blob push semantics would have saved us a considerable amount of diagnosis.

---

Output of docker version

Client:
 Version:     29.7.2
 API version: 1.55

Server:
 Engine:
  Version:     29.7.2
  API version: 1.55
 containerd:
  Version:     v2.3.4
 runc:
  Version:     v1.4.3

Output of docker info (relevant excerpt, failing configuration)

Server Version: 29.7.2
Storage Driver: overlayfs
 driver-type: io.containerd.snapshotter.v1
Cgroup Driver: systemd
Cgroup Version: 2
containerd version: v2.3.4
runc version: v1.4.3
Kernel Version: 6.17.0-35-generic
Operating System: Ubuntu 24.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.3GiB
Live Restore Enabled: false
Firewall Backend: iptables

Additional environment details

Ubuntu 24.04.4 LTS virtual machines, on-premises, no HTTP proxy configured for the daemon, no userns-remap. Before the change, /etc/docker/daemon.json did not exist on either host. The reference host that succeeds is the same engine version, upgraded in place from an earlier release, and therefore still on overlay2.