Multilingual Corpus collection utility tool
I'm currently building a Python utility class that can create a corpus for training a tokenizer with just a few method calls.
For natural languages, it uses Wikipedia and news articles, while for programming languages, it uses GitHub. Users can specify the language, programming language, and the amount of data they need in MB, with the output being a single text file ready to train a tokenizer on.
The utility can also be used to explore how including more non-English languages in a training corpus affects the resulting tokenizer and its inference behavior.
Is this something that could be useful to others learning about tokenization? Has something similar already been made?
Any suggestions or feedback would be greatly appreciated.
Source: karpathy/minbpe