Can `preprocess_data.py` support Huggingface Dataset?
Author: cafeiiCreated Nov 13, 2024Updated Sep 5, 2026
Labelsfeature request
Since there are many datasets in the format of Huggingface datasets, it would be convenient if `preprocess_data.py` can directly preprocess and tokenize from HF datasets.
Source: EleutherAI/gpt-neox