obstore

obstore

The simplest, highest-throughput [^1] Python interface to Amazon S3, Google Cloud Storage, Azure Storage, & other S3-compliant APIs, powered by Rust.

• One interface for all backends with no required Python dependencies.
• Sync and async API with full type hinting.
• Streaming downloads with configurable chunking.
• Streaming uploads from files or async or sync iterators.
• Streaming list, with no need to paginate.
• Automatic multipart uploads for large file objects.
• Automatic credential refresh before expiration.
• File-like object API and fsspec integration.
• Optionally return list results in Apache Arrow format, which is faster and more memory-efficient than materializing Python dicts.
• Zero-copy data exchange between Rust and Python via the buffer protocol.

For Rust developers looking to add object_store support to their own Python packages, refer to pyo3-object_store.

[^1]: Benchmarking is ongoing, but preliminary results indicate roughly 9x higher throughput than fsspec and 2.8x higher throughput than aioboto3 for many concurrent, small, get requests from an async context.

Installation

To install obstore using pip:

pip install obstore

Obstore is on conda-forge and can be installed using conda, mamba, or pixi. To install obstore using conda:

conda install -c conda-forge obstore

Documentation

Full documentation is available on the website.

Head to Getting Started to dig in.