#1321·gpt-neox

Can `preprocess_data.py` support Huggingface Dataset?

Author: cafeiiCreated Nov 13, 2024Updated Sep 5, 2026
Labelsfeature request

Since there are many datasets in the format of Huggingface datasets, it would be convenient if `preprocess_data.py` can directly preprocess and tokenize from HF datasets.