Support cloud storage in load_dataset
Author: lhoestqCreated Nov 22, 2022Updated Sep 17, 2026
Labelsenhancementgood second issue
Would be nice to be able to do
data_files=["s3://..."] # or gs:// or any cloud storage path
storage_options = {...}
load_dataset(..., data_files=data_files, storage_options=storage_options)
The idea would be to use fsspec as in download_and_prepare and save_to_disk.
This has been requested several times already. Some users want to use their data from private cloud storage to train models
related:
https://github.com/huggingface/datasets/issues/3490 https://github.com/huggingface/datasets/issues/5244 forum
Source: huggingface/datasets