Error envelope is appended to a response body that has already started streaming
Description
When a handler fails part way through writing the response body, the dispatcher still calls errcode.ServeJSON:
https://github.com/distribution/distribution/blob/main/registry/handlers/app.go#L719-L726
defer func() {
// Automated error response handling here. Handlers may return their
// own errors if they need different behavior (such as range errors
// for layer upload).
if context.Errors.Len() > 0 {
_ = errcode.ServeJSON(w, context.Errors)
app.logError(context, context.Errors)
} else if ...At that point the status line and headers are already on the wire. net/http discards the second WriteHeader and logs http: superfluous response.WriteHeader call, but the JSON body is still encoded onto the connection, so the client receives the partial payload with an error object appended to it.
Two consequences:
- The bytes on the wire do not match the response. A client reading a blob gets
Nbytes of layer data followed by{"errors":[...]}, under a200status and aContent-Lengthit never reaches. Digest verification catches this for content-addressed pulls, so it usually surfaces as a confusing digest mismatch or unexpected EOF rather than as the actual upstream error. - The recorded status and content type no longer describe what was sent.
errcode.ServeJSONsetsContent-Type: application/jsonand callsWriteHeader(500). Even thoughnet/httpignores both,instrumentedResponseWriterrecords them, soGetResponseLoggerreportshttp.response.status=500 http.response.contenttype=application/jsonfor a response the client received as200 application/octet-stream.
We see this on a registry running in proxy (pull-through cache) mode against a remote registry, where upstream reads fail part way through a layer. Over one week on a single deployment: 67 requests had already written more than 100 KB of layer data before the envelope went out, the largest 71 MB, all logged as status=500 contenttype=application/json.
Note this is distinct from #4518, which was the same superfluous response.WriteHeader log text coming from the otelhttp instrumentation wrapper and went away with the otelhttp upgrade in #4507. The dispatcher path above is unchanged.
Reproduce
- Run the registry in proxy mode against a remote registry.
- Make an upstream blob read fail after the response body has started, for example by interrupting connectivity to the upstream mid-pull.
- Observe the client receiving truncated layer bytes with a JSON error object appended, and the registry logging
response completed with errorwithhttp.response.status=500while a200was sent.
A unit-level reproduction is straightforward: register a handler that writes a few bytes to w and then appends to ctx.Errors, and assert on the response body.
Expected behavior
Once any body bytes have been written, the error envelope cannot be delivered, so it should not be written at all. The error should be logged and the response left as the truncated body it is. Header-only writes should keep the current behaviour, since a handler that sets its own status but writes no body still wants the envelope (layer upload range errors rely on this).
registry version
Present on main and on v3.1.1.
Additional Info
Happy to send a PR: the fix is a written > 0 check on http.response.written, which instrumentedResponseWriter already exposes through Context.Value.
Source: distribution/distribution