Streaming uploads and downloads across storage clients and storages

apify-client-python is adding streaming request bodies in https://github.com/apify/apify-client-python/pull/1060. After it lands, set_record accepts a file-like object, an iterator of bytes/str chunks, or a streamed HttpResponse, and sends it chunked without buffering. The download direction (stream_record) has been there for a long time.

Crawlee has no equivalent at any layer: not in the KeyValueStoreClient / DatasetClient contracts, not in the KeyValueStore / Dataset frontends, and not in the backends. Passing a file object through today's API also corrupts the record silently on three of the five KVS backends.

#1931 asks for streaming KVS records and is still in solutioning. This issue is the wider investigation it needs: what the interfaces should look like, what each backend can actually do, and what the storages should expose.

Related

• apify/apify-client-python#1060
• #1931
• apify/crawlee#2929

✍️ Drafted by Claude Code