aws-chunked encoding issue while using nginx as https proxy
Environment
> uname -a
Linux toptop 6.4.0-150600.23.47-default #1 SMP PREEMPT_DYNAMIC Thu Apr 3 03:44:04 UTC 2025 (2854fd7) x86_64 x86_64 x86_64 GNU/Linux
> /usr/sbin/nginx -V
nginx version: nginx/1.28.0
built by gcc 7.5.0 (SUSE Linux)
built with OpenSSL 3.1.4 24 Oct 2023
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -m64 -fmessage-length=0 -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'Description
Uploading a file to an S3 bucket via HTTPS NGINX proxy_pass is currently failing due to malformed headers.
> aws s3api put-object --bucket my_bucket --key toto.txt --body toto.txt --endpoint-url https://toto.example.com --ca-bundle toto.example.com.crt
An error occurred (MalformedTrailerError) when calling the PutObject operation: The request contained trailing data that was not well-formed or did not conform to our published schema.With the debug flag enabled, we can observe that aws-chunked is being used as Content-Encoding along with x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER, which does not appear to be fully supported by NGINX.
2025-07-09 20:58:51,863 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/my_bucket/toto.txt
content-encoding:aws-chunked
host:toto.example.com
x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date:20250709T185851Z
x-amz-decoded-content-length:8
x-amz-sdk-checksum-algorithm:CRC64NVME
x-amz-security-token: long_token
x-amz-trailer:x-amz-checksum-crc64nvme
content-encoding;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-security-token;x-amz-trailer
STREAMING-UNSIGNED-PAYLOAD-TRAILERNote that it works with HTTP proxy_pass.
Two workarounds to make it work: 1 - Use aws cli < 1.23 2 - Set environment variable AWS_REQUEST_CHECKSUM_CALCULATION to when_required
> export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
> aws s3api put-object --bucket pbidaultdummybucket --key toto.txt --body toto.txt --endpoint-url https://toto.example.com --ca-bundle toto.example.com.crt
{
"ETag": "\"874f0bc52c9563dda66c96\"",
"ChecksumCRC64NVME": "0cvvWi+LXfc=",
"ChecksumType": "FULL_OBJECT",
"ServerSideEncryption": "AES256"
}It appears that the breaking change originates from AWS CLI version 2.23.0, specifically with the implementation of trailer chunked requests. (See changelog: https://raw.githubusercontent.com/aws/aws-cli/v2/CHANGELOG.rst)
feature:s3: S3 client behavior is updated to always calculate CRC64NVME checksum by default for operations that support it, such as PutObject or UploadPart, or require it, such as DeleteObjects. Checksum behavior can be configured using the when_supported and when_required options - in the shared AWS config file using request_checksum_calculation and as an env variable using AWS_REQUEST_CHECKSUM_CALCULATION. Note: AWS CLI will no longer automatically compute and populate the Content-MD5 header.
AWS cli announcement regarding this: https://github.com/aws/aws-cli/issues/9214
- The bug is reproducible with the latest version of nginx
- The nginx configuration is minimized to the smallest possible to reproduce the issue and doesn't contain third-party modules
nginx configuration
> cat /etc/nginx/conf.d/toto.conf
server {
listen 443 ssl;
server_name toto.example.com;
ssl_certificate /etc/nginx/ssl/toto.example.com.crt;
ssl_certificate_key /etc/nginx/ssl/toto.example.com.key;
location / {
proxy_pass http://192.168.69.100:32708;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
}
}In my case, http://192.168.69.100:32708 is an Envoy HTTP proxy that authenticates to an S3 bucket.
Also see https://github.com/kubernetes/ingress-nginx/issues/13589
Source: nginx/nginx