#778·Nginx

aws-chunked encoding issue while using nginx as https proxy

Author: philippebiCreated Jul 9, 2025Updated Sep 30, 2026
Labelsbugneed more info

Environment

> uname -a
Linux toptop 6.4.0-150600.23.47-default #1 SMP PREEMPT_DYNAMIC Thu Apr  3 03:44:04 UTC 2025 (2854fd7) x86_64 x86_64 x86_64 GNU/Linux

> /usr/sbin/nginx -V
nginx version: nginx/1.28.0
built by gcc 7.5.0 (SUSE Linux) 
built with OpenSSL 3.1.4 24 Oct 2023
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

Description

Uploading a file to an S3 bucket via HTTPS NGINX proxy_pass is currently failing due to malformed headers.

> aws s3api put-object --bucket my_bucket --key toto.txt --body toto.txt --endpoint-url https://toto.example.com --ca-bundle toto.example.com.crt

An error occurred (MalformedTrailerError) when calling the PutObject operation: The request contained trailing data that was not well-formed or did not conform to our published schema.

With the debug flag enabled, we can observe that aws-chunked is being used as Content-Encoding along with x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER, which does not appear to be fully supported by NGINX.

2025-07-09 20:58:51,863 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/my_bucket/toto.txt

content-encoding:aws-chunked
host:toto.example.com
x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date:20250709T185851Z
x-amz-decoded-content-length:8
x-amz-sdk-checksum-algorithm:CRC64NVME
x-amz-security-token: long_token
x-amz-trailer:x-amz-checksum-crc64nvme

content-encoding;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-security-token;x-amz-trailer
STREAMING-UNSIGNED-PAYLOAD-TRAILER

Note that it works with HTTP proxy_pass.

Two workarounds to make it work: 1 - Use aws cli < 1.23 2 - Set environment variable AWS_REQUEST_CHECKSUM_CALCULATION to when_required

> export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
> aws s3api put-object --bucket pbidaultdummybucket --key toto.txt --body toto.txt --endpoint-url https://toto.example.com --ca-bundle toto.example.com.crt
{
    "ETag": "\"874f0bc52c9563dda66c96\"",
    "ChecksumCRC64NVME": "0cvvWi+LXfc=",
    "ChecksumType": "FULL_OBJECT",
    "ServerSideEncryption": "AES256"
}

It appears that the breaking change originates from AWS CLI version 2.23.0, specifically with the implementation of trailer chunked requests. (See changelog: https://raw.githubusercontent.com/aws/aws-cli/v2/CHANGELOG.rst)

feature:s3: S3 client behavior is updated to always calculate CRC64NVME checksum by default for operations that support it, such as PutObject or UploadPart, or require it, such as DeleteObjects. Checksum behavior can be configured using the when_supported and when_required options - in the shared AWS config file using request_checksum_calculation and as an env variable using AWS_REQUEST_CHECKSUM_CALCULATION. Note: AWS CLI will no longer automatically compute and populate the Content-MD5 header.

AWS cli announcement regarding this: https://github.com/aws/aws-cli/issues/9214

  • The bug is reproducible with the latest version of nginx
  • The nginx configuration is minimized to the smallest possible to reproduce the issue and doesn't contain third-party modules

nginx configuration

> cat /etc/nginx/conf.d/toto.conf 
server {
    listen 443 ssl;
    server_name toto.example.com;

    ssl_certificate     /etc/nginx/ssl/toto.example.com.crt;
    ssl_certificate_key /etc/nginx/ssl/toto.example.com.key;

    location / {
        proxy_pass         http://192.168.69.100:32708;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;
    }
}

In my case, http://192.168.69.100:32708 is an Envoy HTTP proxy that authenticates to an S3 bucket.

Also see https://github.com/kubernetes/ingress-nginx/issues/13589