Tiktoken educational BPE trainer takes long time to train with vocab size 30k
Author: sagorbrurCreated May 19, 2024Updated Aug 31, 2026
Hi, I am trying to train Tiktoken on a custom dataset (size 15 GB) with 30k vocab size. It seems it will take a long time to finish. 1 vocab update took almost 8 hours. Any suggestion to make it faster? Thanks in advance.
Source: openai/tiktoken