用于 Python、Go、C++ 和 JavaScript 的非贪心子词分词器和词汇训练器
UPDATE: Benchmark results from pretraining 16 language models on different tokenizers.
TokenMonster is an ungreedy subword tokenizer and vocabulary generator, enabling language models to run faster, cheaper, smarter and generate longer streams of text.
Large and sub-optimal vocabularies lead to the waste of computational and memory resources in language models. By switching to TokenMonster, you can potentially achieve the same or better performance with a vocabulary that is less than a quarter of the size.
TokenMonster can train and generate an optimal vocabulary on a 1 GB dataset within 24 hours on a typical desktop. 442 pretrained vocabularies are provided, as well as tools to train your own vocabularies & implementations in Go, Python & Javascript for tokenization and detokenization using the pretrained or your own vocabularies.
You can test TokenMonster in your browser here, tokenizing live in native Javascript.
TokenMonster is a novel approach to tokenization with broad-ranging use potential, but its primary motivation is to improve the training, inference and context-length of large language models. By using a more optimal vocabulary and ungreedy tokenization algorithm, text can be represented with 37.5% fewer tokens at the same vocabulary size compared to other modern tokenizing methods, increasing the speed of inference, training and the length of text. And/or the vocabulary size can be reduced by 75% or more, freeing resources that can be used to make the model smarter and faster.
You can also import existing vocabularies from other tokenizers, allowing you to take advantage of TokenMonster's fast, ungreedy tokenization whilst still using the existing vocabulary your model was trained for. TokenMonster vocabularies for GPT2 Tokenizer and LLaMa Tokenizer are included.
unfiltered, clean, balanced, consistent, strict442 vocabularies are planned or have already been built. Download them from Hugging Face, or in the Python library you can simply specify them by name and they'll be downloaded automatically. (Note: the pretrained vocabularies are still being trained, check here to see which are currently available.)
code english englishcode fiction1024 2048 4096 8000 16000 24000 32000 40000 50256 65536 100256unfiltered clean balanced consistent strictnocapcodev1Examples: fiction-24000-strict-v1 code-4096-clean-nocapcode-v1
Usage:
import tokenmonster
vocab = tokenmonster.load("englishcode-32000-consistent-v1")
tokens = vocab.tokenize("This is a test.")There are 2 additional pre-built vocabularies: gpt2 and llama. These are imports of GPT2 Tokenizer and LLaMa Tokenizer from Hugging Face Transformers into TokenMonster. The tokens and IDs are identical, however they do not always tokenize the text in exactly the same way. For example, LLaMa Tokenizer on Hugging Face tokenizes " decoded" as dec oded, whilst TokenMonster tokenizes [correctly] to decode d. TokenMonster trained vocabularies are massively more efficient, so only use gpt2 and llama if you have to. The scripts used to import them into TokenMonster are here.
vocab = tokenmonster.load("gpt2")All the optimization modes are lossless. The stricter the optimization mode (higher number), the more tokens will be used to tokenize the same text, but it'll be much easier for the language model to learn because the grammar is simpler. Less strict (lower number), more text can be represented with fewer tokens, but the language model will have to learn a more complicated grammar.
0 unfiltered allows the training process to freely determine the tokens. clean is preferred in almost every case, because unfiltered tends to result in overfitting, especially for code as it results in tokens for things like \n\t\t\t\tif (. Use unfiltered for tokenizing language or data that does not use spaces as word boundaries.
1 clean introduces filters to avoid overfitting. It forces the vocabulary to begin words with a space, and limits the way in which whitespace can be combined with other characters.
2 balanced prioritizes whole words and attempts to dissuade the vocabulary from doing things that are difficult to learn.
3 consistent is a looser version of strict. It aims to limit the number of different tokens that can represent the same word or phrase, and doesn't allow for open-close delimeters to be combined with words or each other. Numbers also become limited to fewer variants.
4 strict aims to have only 1 token per word, no matter how it is encoded. For example However, however, and HOWEVER! will all use the same however token, in combination with other tokens that indicate it's spacing and capitalization.
View TokenMonster Vocabulary Comparison, to see a line chart of the relationship between vocab size, optimization mode and characters/token. From this chart I can stipulate the rule of thumb that every doubling of vocabulary size inscreases the characters/token by 0.5. This pattern starts from vocab size 4096 and consistent up to 100256.
It's tempting to use large vocabularies, which has been norm, but you can see on the TokenMonster Tester and Interactive Benchmark that reducing the vocabulary by 50 - 75% can often result in only a relatively minor increase to the number of tokens required to tokenize it. Even the very general englishcode vocabularies, which are for all intents and purposes multi-lingual, do very well at vocab size 24000. Story or article writing models can go as low as 4096 vocabulary size and still tokenize at 4 characters per token.
TokenMonster works well with small vocabularies because it's using an optimal selection process. In most cases it's simply not necessary to use vocabulary sizes greater than 32000, unless it's a multi-lingual vocabulary. More is not better. Using a vocabulary that is excessively large can lead to inefficient usage of embeddings, not to mention an over-complicated grammar. The embeddings for all those unneeded tokens occupy memory and computational resources that could be used more efficiently.
In my opinion, the 100K vocabulary size is excessive and wasteful, unless your aim is to support at least three languages in the same vocabulary. With a 100K size, you have "spare" tokens. By "spare", I mean that the vocabulary starts assigning tokens to lengthy, specific sequences like "limitations under the License" and "#### According to", suggesting that the vocabulary has reached its optimal size and is now just compressing frequently occurring strings.
My advice is to find the smallest vocabulary size that meets your requirements. With this, you can either be content with a smaller, faster model, or opt to augment the size of the embeddings accordingly, or find a balance between the two.
In regards to optimization modes, strict is the one to go for if your model is limited by its size or largely undertrained. If it's a small model that isn't particularly smart, and you want to get the most out of it, choose strict because it'll probably result in a smarter model given that the simpler grammar is quicker to learn (words, punctuation and modifiers are all separate tokens.) On the other hand, if you're training something serious with enough training data so that each token is exposed to a variety of contexts in order to learn it's more complex grammar, you probably want to go for clean or balanced.
strict performs very well with longform natural text, such as novels and articles, but it's too strict for code. consistent will give the best balance of consistency for tokenizing code whilst keeping the grammar simple. balanced and clean are excellent at compressing code into fewer tokens, but this comes with the trade-off of more complex grammar. That said, a smaller vocabulary implies a simpler grammar (less possible combinations), so it may be in your interest to aim for balanced with a fairly small vocabulary size, such as 16000. All of this you can determine by playing around with TokenMonster Tester.
Capcode is an alternative encoding for uppercase in UTF-8 text, supporting all UTF-8 characters. It's completely lossless, changing the way in which capital letters are encoded so they can share tokens with lowercase letters but without losing any information. In theory, capcode makes it easier for a model to learn the meaning of words. Additionally, capcode makes for more efficient tokenization because it frees up so many tokens that would otherwise be used for uppercase variants of already existing lowercase tokens.
TokenMonster is designed to be plug-and-play, taking care of normalization concerns for you. UTF-8 and UTF-16 vocabularies are automatically NFD normalized and encoded Little Endian regardless of architecture. When tokenizing, the exact same transformations are applied transparently, so you can pass a string to either UTF-8 or UTF-16 vocabularies, with or without capcode, and on either Little or Big Endian architecture, and it will b
暂无开放 Issues，或尚未同步最近议题。