[Bug] Unconditional SHA-256 ChecksumAlgorithm on PutObject silently corrupts objects on OTC/Huawei OBS - fix #22003 merged to main but never released

Describe the bug

Since #21848 / #21849 (first released in v3.6.11 and v3.7.0), S3ObjectClient.PutObject sets ChecksumAlgorithm: types.ChecksumAlgorithmSha256 unconditionally on every PutObject, without a pre-computed checksum value:

https://github.com/grafana/loki/blob/v3.7.6/pkg/storage/chunk/client/aws/s3_storage_client.go#L543

When ChecksumAlgorithm is set but no value is supplied, AWS SDK Go v2 switches to trailing checksum mode and wraps the body in aws-chunked framing:

content-encoding: aws-chunked
x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-decoded-content-length: <n>
x-amz-trailer: x-amz-checksum-sha256

On Open Telekom Cloud / Huawei OBS (S3-compatible) this does not fail loudly the way it does on NetApp ONTAP (400 InvalidArgument, #21926) or OpenStack Swift (501 aws-chunked is not supported, #21791).

OBS accepts the request with 200 OK, but the stored object is not readable back. What we observed directly: no client-side error at write time, and the same object failing to gunzip on the next read.

Our working explanation - and we want to be explicit that this is inference from the failure signature, not something we confirmed by byte-inspecting the stored object - is that OBS does not strip the aws-chunked framing before persisting the body. We could not confirm it from the client side because OBS returns no response checksum, so SDK v2 skips validation entirely:

SDK WARN: Response has no supported checksum. Not validating response payload.

Whatever the precise mechanism inside OBS, the externally visible behaviour is the same and is what matters here: a write that reports success produces an object that cannot be read back.

The result is silent corruption of TSDB index objects. Nothing fails at write time - no error, no retry, no metric. The damage only surfaces when a component reads the objects back:

downloaded file .../loki_index_XXXXX/....tsdb.gz
sync failed, retrying it   err="gzip: invalid header"
failed to initialize table ... error running loki: gzip: invalid header

loki-backend-0 then enters a permanent CrashLoopBackOff (exit code 1).

This is not recoverable by rolling back. The corrupt objects stay in the bucket, and they break store initialisation for the healthy version too - so the StatefulSet rollback deadlocks in the same crash loop. Recovery required manually deleting the corrupt index objects from the bucket (aws s3 rm s3://<bucket>/index/<table>/...) before a known-good version would start. Chunk objects were unaffected in our case, but we have no way to prove they are generally safe.

To Reproduce

1. Run Loki v3.6.11 or newer (any 3.6.11-3.6.15, 3.7.0-3.7.6) with storage_config.aws pointed at an OTC/Huawei OBS endpoint. Virtual-hosted addressing (OBS has forbidden path-style since 2023-12-30, so this is the only supported mode). Object Lock is NOT enabled on the bucket - the checksum is attached anyway.
2. Ingest logs and let the compactor write/upload TSDB index tables. Everything reports success: PutObject returns 200, flush counters increase, level=error count stays at 0.
3. Restart the backend target (or wait for any pod that re-initialises the index from object storage).
4. Observe gzip: invalid header on the downloaded .tsdb.gz files and a CrashLoopBackOff.

For contrast, we ran the identical chart, identical config and the identical bucket on v3.6.7 and v3.6.8 for 20h+ with the same index tables and saw zero such errors. The only variable changed was the Loki binary.

Expected behavior

PutObject should either succeed and store a byte-identical object, or fail loudly. Silently storing a corrupted object is the worst possible outcome, because the blast radius grows for as long as the problem goes unnoticed.

Concretely, two asks:

1. Please backport #22003 to release-3.6.x and release-3.7.x.

#22003 ("fix(storage): Pre-compute SHA-256 to avoid aws-chunked on PutObject") already solves exactly this: it computes the SHA-256 up front and sends it as a plain x-amz-checksum-sha256 header instead of a streaming trailer. It was merged to main on 2026-05-29.

However, it is not present in any released version:

There is no release-3.8.x branch, and the newest releases in both active lines - v3.6.15 and v3.7.6, both published 2026-08-06 - still contain the unfixed code. So the fix has been merged but unreachable for roughly three months, and every currently released version from 3.6.11 onward carries a silent-corruption bug against at least one major S3-compatible provider.

2. Please make the Object Lock checksum conditional rather than unconditional.

Even with #22003 in place, attaching a SHA-256 to every PutObject is only required for buckets with Object Lock enabled, which is what #20088 asked for. Buckets without Object Lock pay the cost (and the compatibility risk) for a feature they do not use.

A config option - e.g. storage_config.aws.checksum_algorithm: sha256|none - would let operators opt out when a vendor's S3 implementation mishandles the header, without having to wait for a release. Right now there is no escape hatch in configuration at all: the only lever is pinning to <= 3.6.10.

Related question: is there a supported workaround for affected users on current releases? Setting AWS_REQUEST_CHECKSUM_CALCULATION=when_required does not look applicable here, because the algorithm is set explicitly on the request rather than coming from the SDK default - but we have not verified that, and would rather ask than guess.

Relationship to #21926

#21926 (open) covers the same root cause - #21848/#21849 - but a different and much more visible failure mode: a hard 400 InvalidArgument at write time. That fails safe: the operator sees it immediately, nothing is written, and a rollback works.

The OBS behaviour is the opposite: writes are accepted, the data is corrupt, and detection is delayed until the next read - by which point the bucket contains corrupt objects that also break the version you would roll back to. We think that is worth tracking separately, because it changes both the severity assessment and the recovery procedure, but we are happy for this to be folded into #21926 if maintainers prefer.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm (grafana/loki chart, simple-scalable / SSD mode)
• Object storage: Open Telekom Cloud (Huawei) OBS, S3-compatible endpoint, virtual-hosted-style addressing, SigV4, no Object Lock on the affected bucket
• Affected versions: 3.6.11 - 3.6.15, 3.7.0 - 3.7.6
• Last known-good: 3.6.8 (chart 7.1.0), verified over a 20h+ soak against the same bucket

Screenshots, Alloy config, or terminal output

Backend crash loop while initialising the index from OBS:

level=info  msg="downloaded file" table-name=loki_index_XXXXX file-name=....tsdb.gz
level=error msg="sync failed, retrying it" err="gzip: invalid header"
level=error msg="failed to initialize table" table-name=loki_index_XXXXX
error running loki: gzip: invalid header

SDK warning present on every response from OBS, which is why the corruption is invisible to the client:

SDK 2026/XX/XX WARN Response has no supported checksum. Not validating response payload.

Impact on upgrades

This currently pins us to Loki <= 3.6.10 (chart 7.1.0 / Loki 3.6.8, which is the newest published chart and happens to predate the break). We cannot pick up any fix - including security fixes - shipped in 3.6.11+ or 3.7.x without reintroducing silent index corruption. A backport of #22003 into either release line would unblock that immediately.