#195·tiktoken

Very slow for inputs like "a" * 100000

Author: vvolhejnCreated Sep 18, 2023Updated May 24, 2026

I've noticed that Tiktoken is really slow for strings of repeated characters like "a" * 100_000. Interestingly, when you add spaces, like "a " * 50_000, the performance is orders of magnitude better:

Screenshot 2023-09-18 at 09 56 21

Is this a bug or a fundamental property of BPE?

My Tiktoken version is 0.5.0.