#299·tiktoken

Tiktoken educational BPE trainer takes long time to train with vocab size 30k

Author: sagorbrurCreated May 19, 2024Updated Aug 31, 2026

Hi, I am trying to train Tiktoken on a custom dataset (size 15 GB) with 30k vocab size. It seems it will take a long time to finish. 1 vocab update took almost 8 hours. Any suggestion to make it faster? Thanks in advance.

https://github.com/openai/tiktoken/blob/c0ba74c238d18b4824c25f3c27fc8698055b9a76/tiktoken/_educational.py#L117