minbpe · Issues· 59 open

Multilingual Corpus collection utility tool

train() raises ValueError: max() arg is an empty sequence on empty or whitespace-only input

encode(..., allowed_special="none_raise") raises AssertionError instead of ValueError

Python API with C extensions for faster training and encoding

get_gpt4_merges()

Simpler and faster encoding

Notebook Issue In Google Colab

What to support GPT-4O tokenizer？

LLM is worse at non-English languages

Question about Encoder Logic

One problem in the annotations of `test_wikipedia_example` in the `tests/test_tokenizer` file

counting pairs is inaccurate for repeating tokens?

Instead of finding the one pair with the highest frequency and merging it at each step, do the highest N pairs

LLM as calc

OSS-Fuzz Integration