s3proxy

S3Proxy

S3Proxy implements the S3 API and proxies requests, enabling several use cases:

• translation from S3 to Google Cloud, Microsoft Azure, OpenStack Swift, and SFTP
• testing without Amazon by using the local filesystem
• extension via middlewares
• embedding into Java applications

Usage with Docker

Docker Hub hosts a Docker image and has instructions on how to run it.

Usage with Kubernetes

Reference manifests show how to wire the Docker image into Kubernetes, including health probes, graceful shutdown, and Secret-based credentials. Helm users may prefer the third-party s3proxy-chart.

Usage without Docker

Users can download releases from GitHub. Developers can build the project by running mvn package which produces a binary at target/s3proxy. S3Proxy requires Java 17 or newer to run.

Configure S3Proxy via a properties file. An example using the local file system as the storage backend with anonymous access:

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

First create the filesystem basedir:

mkdir /tmp/s3proxy

Next run S3Proxy. Linux and Mac OS X users can run the executable jar:

chmod +x s3proxy
s3proxy --properties s3proxy.conf

Windows users must explicitly invoke java:

java -jar s3proxy --properties s3proxy.conf

Finally test by creating a bucket then listing all the buckets:

$ curl --request PUT http://localhost:8080/testbucket

$ curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>[email protected]</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

Usage with Java

Maven Central hosts S3Proxy artifacts and the wiki has instructions on Java use.

Supported storage backends

• aws-s3
• azureblob
• filesystem (on-disk storage)
• google-cloud-storage
• openstack-swift (Keystone v3 only)
• sftp (SFTP storage via Apache MINA SSHD)
• transient (in-memory storage)

See the wiki for examples of configurations.

Backend credentials

jclouds.identity and jclouds.credential name the enduring half of a backend credential. Credentials that expire carry a third part, which jclouds.session-token names: an AWS session token, an Azure shared access signature, a Google OAuth 2.0 access token, or a Keystone token.

jclouds.provider=aws-s3
jclouds.identity=ASIAIOSFODNN7EXAMPLE
jclouds.credential=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
jclouds.session-token=IQoJb3JpZ2luX2VjEExample

A token written into the properties file serves no longer than the token itself does. A deployment that rotates them embeds S3Proxy and hands BlobStores.create a Supplier<Credentials> instead: a store asks it again for every request it signs rather than keeping the first answer, so an assumed role, a reissued signature, or a refreshed access token takes effect without the store being rebuilt.

Naming no credentials at all defers to the chain each backend's own tools read -- environment, profile, container, and instance role for aws-s3, DefaultAzureCredential for azureblob, and application default credentials for google-cloud-storage -- every one of which renews what it finds.

Three credentials are read only when the store is built, because nothing about them expires: an Azure account key, which the client copies out when it is created; a Google service account key, which mints and renews access tokens of its own; and the sftp password, presented once when the connection opens. openstack-swift re-reads the supplier whenever the Keystone token it holds nears its expiry.

Assigning buckets to backends

S3Proxy can be configured to assign buckets to different backends with the same credentials. The configuration in the properties file is as follows:

s3proxy.bucket-locator.1=bucket
s3proxy.bucket-locator.2=another-bucket

In addition to the explicit names, glob syntax can be used to configure many buckets for a given backend.

A bucket (or a glob) cannot be assigned to multiple backends.

Middlewares

S3Proxy can modify its behavior based on middlewares:

• bucket aliasing
• bucket prefix scoping
• bucket locator
• eventual consistency modeling
• large object mocking
• latency
• read-only
• regex rename blobs
• sharded backend containers
• storage class override
• user metadata replacer
• no cache override

SSL Support

S3Proxy can listen on HTTPS by setting the secure-endpoint and configuring a keystore. You can read more about how configure S3Proxy for SSL Support in the dedicated wiki page with Docker, Kubernetes or simply Java.

Limitations

S3Proxy has broad compatibility with the S3 API, however, it does not support:

• ACLs other than private and public-read, including the x-amz-grant-* headers and any grant naming a specific grantee
• BitTorrent hosting
• bucket inventory, analytics, and metrics configuration
• bucket lifecycle configuration
• bucket logging
• bucket notification configuration
• bucket policies
• bucket policy status
• bucket replication
• CORS bucket operations like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration (see below).
• hosting static websites
• object lock, including legal hold and retention
• object ownership controls
• object tagging
• paginating ListParts with part-number-marker
• public access block
• reading a single part of a multipart object with partNumber, unless the object has only one part
• requester pays buckets
• restoring archived objects
• select object content
• transfer acceleration
• x-amz-expected-bucket-owner

S3Proxy emulates the following operations:

• multi-part upload on the filesystem and transient backends, which store a stub object to carry the metadata
• object and bucket owners, which are always the same synthetic user

Some limitations depend on the storage backend:

The backends that do version objects come by it differently. aws-s3 passes the requests through to a service that already versions, while google-cloud-storage translates S3 versioning onto GCS object generations, emulating delete markers as zero-byte generations; because GCS versioning is simply on or off, it refuses the Suspended state. transient implements versioning itself, keeping every version but the current one as a hidden object. filesystem shares that implementation and declines it: its versions would outlive the process, settling a layout on disk that S3Proxy would owe its users from then on.

Server-side encryption divides aws-s3 and transient from most of the rest, and divides the two nio2 stores on the same line. aws-s3 forwards the header families -- SSE-S3, SSE-KMS and SSE-C alike -- to a service that encrypts, while transient answers them itself: what it holds rests in a filesystem that dies with the process, so an object's encryption is the headers and nothing besides, which is all S3 lets a caller observe of SSE-S3 anyway. SSE-C has an observable half beyond the headers -- an object stored under a customer's key answers only to that key -- and transient enforces it, keeping the key's MD5 and never the key. Unlike S3, S3Proxy accepts SSE-C over plain HTTP, its TLS commonly terminating elsewhere. filesystem writes to a real disk, where reporting AES256 over plaintext someone could go and read would be a claim S3Proxy has no business making, so it refuses every one of these headers instead. azureblob stands halfway in: Azure encrypts everything at rest and says so on each response, which the store relays as SSE-S3 -- an object's AES256 is Azure's answer, not S3Proxy's assertion -- while SSE-C and SSE-KMS are refused rather than silently accepted.

Bucket default encryption, the ?encryption subresource, rides the same divide. aws-s3 passes the configuration through, transient keeps it and rests unadorned writes under it -- a write naming its own encryption still wins -- and every other backend answers NotImplemented. A bucket never given a configuration answers the 404 S3 spells ServerSideEncryptionConfigurationNotFoundError rather than the account-wide AES256 default modern AWS invents, which is also what the Ceph s3-tests expect of it.

The filesystem backend stores a key as a path, so on Windows a key holding a backslash is refused with 400 InvalidArgument. Windows reads that character as a directory separator and no file name can hold one, so the key back\slash.txt would come back from a listing as back/slash.txt and answer for whoever wrote that key instead -- two S3 keys on one object. Keys are spelled with forward slashes on every host, whatever the filesystem underneath.

aws-s3, azureblob and google-cloud-storage copy a part on the server, in one request and w