blobStatter.stat() method in client expects HEAD requests to always contain a content-length (which is not RFC compliant)
Author: frittenthekeCreated Jun 11, 2026Updated Aug 9, 2026
Description
We observed issues with our own pull-through-cache using Docker Distribution 3.0.0 and pulling from registry.istio.io : The registry was throwing 500s with the following in the error / access log:
time="2026-06-11T07:15:50.764104349Z" level=error msg="response completed with error" err.code=unknown err.detail="missing content-length header for request: https://registry.istio.io/v2/release/proxyv2/blobs/sha256:d276db19ce85199de06ccbf305efb5bd294c7f1bf4f2b76d08d294fc2b044fb8%22 err.message="unknown error" go.version=go1.23.7 http.request.host=oci-cache-istio.REDACTED http.request.id=90e2004f-bebf-4d6f-8df3-211040e8c194 http.request.method=GET http.request.remoteaddr=REDACTED http.request.uri="/v2/release/proxyv2/blobs/sha256:d276db19ce85199de06ccbf305efb5bd294c7f1bf4f2b76d08d294fc2b044fb8?ns=registry.istio.io" http.request.useragent=containerd/2.1.6+bottlerocket http.response.contenttype=application/json http.response.duration=497.048332ms http.response.status=500 http.response.written=235 instance.id=8e82a078-3cab-4503-a7f6-8f7a895dc2e8 service=registry vars.digest="sha256:d276db19ce85199de06ccbf305efb5bd294c7f1bf4f2b76d08d294fc2b044fb8" vars.name=release/proxyv2 version=3.0.0Following the error string into the code, we found this section https://github.com/distribution/distribution/blob/v3.0.0/internal/client/repository.go#L879-L884 expecting a content-length header.
Comparing registry.istio.io with gcr.io I noticed that there is no content-length header anymore:
curl --head -v https://registry.istio.io/v2/release/proxyv2/blobs/sha256:f96e152c1cffa8366b93b5933d1724f7e346f7502a629d30239a42f9e6d4b1ac
* Host registry.istio.io:443 was resolved.
* IPv6: 2606:4700:3032::6815:55f, 2606:4700:3031::ac43:8542
* IPv4: 172.67.133.66, 104.21.5.95
* Trying [2606:4700:3032::6815:55f]:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* SSL Trust Anchors:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Unknown (25):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519MLKEM768 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
* subject: CN=istio.io
* start date: Apr 13 19:32:21 2026 GMT
* expire date: Jul 12 20:32:19 2026 GMT
* issuer: C=US; O=Google Trust Services; CN=WE1
* Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
* Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
* Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* subjectAltName: "registry.istio.io" matches cert's "registry.istio.io"
* OpenSSL verify result: 0
* SSL certificate verified via OpenSSL.
* Established connection to registry.istio.io (2606:4700:3032::6815:55f port 443) from 2a0a:a543:e353:1:3e56:77ac:9912:8f85 port 60654
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://registry.istio.io/v2/release/proxyv2/blobs/sha256:f96e152c1cffa8366b93b5933d1724f7e346f7502a629d30239a42f9e6d4b1ac
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: registry.istio.io]
* [HTTP/2] [1] [:path: /v2/release/proxyv2/blobs/sha256:f96e152c1cffa8366b93b5933d1724f7e346f7502a629d30239a42f9e6d4b1ac]
* [HTTP/2] [1] [user-agent: curl/8.20.0]
* [HTTP/2] [1] [accept: */*]
> HEAD /v2/release/proxyv2/blobs/sha256:f96e152c1cffa8366b93b5933d1724f7e346f7502a629d30239a42f9e6d4b1ac HTTP/2
> Host: registry.istio.io
> User-Agent: curl/8.20.0
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200
HTTP/2 200
< date: Thu, 11 Jun 2026 14:07:22 GMT
date: Thu, 11 Jun 2026 14:07:22 GMT
< content-type: text/html
content-type: text/html
< cf-ray: a0a12e835a357181-DUS
cf-ray: a0a12e835a357181-DUS
< cf-cache-status: DYNAMIC
cf-cache-status: DYNAMIC
< server: cloudflare
server: cloudflare
< vary: Accept-Encoding
vary: Accept-Encoding
< alt-svc: h3=":443"; ma=86400
alt-svc: h3=":443"; ma=86400
< docker-distribution-api-version: registry/2.0
docker-distribution-api-version: registry/2.0
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< x-gcr-using-artifact-registry: true
x-gcr-using-artifact-registry: true
< x-xss-protection: 0
x-xss-protection: 0
< report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=Fsbg28yRmsUwvG9lOe9XLd1J0G3gIR5W2j2LWEoWtTpMePSKN%2Ff5IDQfAEpJLNQLbs42bPoZqnBHPtV8R21V6aifgfJ1rCiLtZTMozqu%2FoPkMK%2Bn0pa4MfOLByfe253s4ESCNHdHlBAeFLBdioqiGw%3D%3D"}]}
report-to: {"group":"cf-nel","max_age":604800,"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v4?s=Fsbg28yRmsUwvG9lOe9XLd1J0G3gIR5W2j2LWEoWtTpMePSKN%2Ff5IDQfAEpJLNQLbs42bPoZqnBHPtV8R21V6aifgfJ1rCiLtZTMozqu%2FoPkMK%2Bn0pa4MfOLByfe253s4ESCNHdHlBAeFLBdioqiGw%3D%3D"}]}
< nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
nel: {"report_to":"cf-nel","success_fraction":0.0,"max_age":604800}
<
* Connection #0 to host registry.istio.io:443 left intactvs.
curl -v --head https://gcr.io/v2/istio-release/proxyv2/blobs/sha256:f96e152c1cffa8366b93b5933d1724f7e346f7502a629d30239a42f9e6d4b1ac
* Trying [2a00:1450:400c:c00::52]:443...
* Host gcr.io:443 was resolved.
* IPv6: 2a00:1450:400c:c00::52
* IPv4: 173.194.76.82
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* SSL Trust Anchors:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519MLKEM768 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
* subject: CN=*.googlecode.com
* start date: May 18 18:37:06 2026 GMT
* expire date: Aug 10 18:37:05 2026 GMT
* issuer: C=US; O=Google Trust Services; CN=WR2
* Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha384WithRSAEncryption
* subjectAltName: "gcr.io" matches cert's "gcr.io"
* OpenSSL verify result: 0
* SSL certificate verified via OpenSSL.
* Established connection to gcr.io (2a00:1450:400c:c00::52 port 443) from 2a0a:a543:e353:1:3e56:77ac:9912:8f85 port 37658
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://gcr.io/v2/istio-release/proxyv2/blobs/sha256:f96e152c1cffa8366b93b5933d1724f7e346f7502a629d30239a42f9e6d4b1ac
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: gcr.io]
* [HTTP/2] [1] [:path: /v2/istio-release/proxyv2/blobs/sha256:f96e152c1cffa8366b93b5933d1724f7e346f7502a629d30239a42f9e6d4b1ac]
* [HTTP/2] [1] [user-agent: curl/8.20.0]
* [HTTP/2] [1] [accept: */*]
> HEAD /v2/istio-release/proxyv2/blobs/sha256:f96e152c1cffa8366b93b5933d1724f7e346f7502a629d30239a42f9e6d4b1ac HTTP/2
> Host: gcr.io
> User-Agent: curl/8.20.0
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200
HTTP/2 200
< content-length: 4445
content-length: 4445
< docker-distribution-api-version: registry/2.0
docker-distribution-api-version: registry/2.0
< server: Docker Registry
server: Docker Registry
< vary: Accept-Encoding
vary: Accept-Encoding
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< x-gcr-using-artifact-registry: true
x-gcr-using-artifact-registry: true
< x-xss-protection: 0
x-xss-protection: 0
< date: Thu, 11 Jun 2026 14:07:44 GMT
date: Thu, 11 Jun 2026 14:07:44 GMT
< content-type: text/html
content-type: text/html
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
* Connection #0 to host gcr.io:443 left intact
The relevant RFC https://datatracker.ietf.org/doc/html/rfc9110#section-8.6 does not force a server to send content-length with HEAD requests:
[...] A server MAY send a Content-Length header field in a response to a HEAD request (Section 9.3.2); a server MUST NOT send Content-Length in such a response unless its field value equals the decimal number of octets that would have been sent in the content of a response if the same request had used the GET method. [...]
Also using a conditional request would not help ... as there content-length is also not required:
[...] A server MAY send a Content-Length header field in a 304 (Not Modified) response to a conditional GET request (Section 15.4.5); a server MUST NOT send Content-Length in such a response unless its field value equals the decimal number of octets that would have been sent in the content of a 200 (OK) response to the same request. [...]
If you change this part of the code checking if something as changed upstream (which likely can only be 404, as the layer digest is content-adressing, right?) maybe it makes sense to even switch to using conditional requests (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/304#304_response_to_conditional_requests)?
Reproduce
- Start docker registry with
proxy:
remoteurl: "https://registry.istio.io" (they will likely fix this soon!!)
- try pulling e.g.
registry.istio.io/release/proxyv2:1.30.1
Expected behavior
No response
registry version
3.0.0 (but code did not change)
Additional Info
- Istio (registry.istio.io) is apparently using https://github.com/howardjohn/registry-redirector/
Source: distribution/distribution