#521·tiktoken

tiktoken 0.12 cannot handle qwen3.5 in huggingface

Author: jslhclCreated Apr 8, 2026Updated May 25, 2026

It looks tiktoken 0.12 cannot handle the tokenizer of qwen3.5 in huggingface

The merges.txt has the length of 247587, the total vocabulary size is 248077, the length of special vocab is 33. so that in function data_gym_to_mergeable_bpe_ranks in load.py, it will hit 'assert bpe_ranks == encoder_json_loaded', as the length of bpe_ranks is now 247587 + 256, which is not equal to the length of encoder_json_loaded (248077 - 33)