#5281·datasets

Support cloud storage in load_dataset

Author: lhoestqCreated Nov 22, 2022Updated Sep 17, 2026
Labelsenhancementgood second issue

Would be nice to be able to do

data_files=["s3://..."]  # or gs:// or any cloud storage path
storage_options = {...}
load_dataset(..., data_files=data_files, storage_options=storage_options)

The idea would be to use fsspec as in download_and_prepare and save_to_disk.

This has been requested several times already. Some users want to use their data from private cloud storage to train models

related:

https://github.com/huggingface/datasets/issues/3490 https://github.com/huggingface/datasets/issues/5244 forum