AutoTokenizer with tokenizer_class=LlamaTokenizerFast silently builds a sentencepiece pipeline for a byte-level BPE tokenizer.json, dropping all CJK input

System Info

transformers: 5.8.1 tokenizers: 0.22.2 Python 3.10.12 platform: Linux x86_64, inside Docker

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer

# Repo: deepseek-ai/DeepSeek-R1-0528-Qwen3-8B
# tokenizer_config.json: "tokenizer_class": "LlamaTokenizerFast", "legacy": true,
#                        "sp_model_kwargs": {}, "model_max_length": 131072
# tokenizer.json:        {"model": {"type": "BPE"}, "decoder": {"type": "ByteLevel"}},
#                        vocab 151643, byte-level glyph keys such as "Ċ" (U+010A)
tok = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-0528-Qwen3-8B")

print(type(tok).__name__, tok.is_fast, getattr(tok, "legacy", None))
print(tok.backend_tokenizer.pre_tokenizer)
print(tok.backend_tokenizer.decoder)
print("cjk  :", tok.encode("嗯，"))
print("ascii:", tok.encode("hello world"), repr(tok.decode(tok.encode("hello world"))))

Expected behavior

The backend tokenizer should use the pipeline serialized in tokenizer.json, i.e. a ByteLevel pre-tokenizer and a ByteLevel decoder — the same thing you get from PreTrainedTokenizerFast(tokenizer_file=".../tokenizer.json"). At minimum, if the class named in tokenizer_config.json cannot be reconciled with the serialized tokenizer, loading should warn or raise loudly instead of silently building something else.

pre_tokenizer -> ByteLevel(...)
encode("嗯，")            -> non-empty ids
encode("hello world")    -> round-trips including the space

A sentencepiece-style pipeline is constructed and the tokenizer.json pipeline is effectively discarded. No warning or error is emitted:

class 'LlamaTokenizer' fast True legacy True
pre_tokenizer -> Metaspace(replacement="▁", prepend_scheme=always, split=False)
decoder -> Sequence([Replace("▁" -> " "), ByteFallback(), Fuse(), Strip(...)])
cjk  : []
ascii: [71, 95292] -> decode -> 'helloworld'      # the space is lost as well

All non-ASCII input is silently dropped, and decode() returns the raw byte-level glyph strings ("Ċ", "åĹ¯") instead of text.

• Changing only "tokenizer_class" to "Qwen2Tokenizer" (and removing "legacy" / "sp_model_kwargs") in tokenizer_config.json makes the very same tokenizer.json load correctly, so the serialized tokenizer is not at fault:

  复制

  type(tok).__name__      -> 'Qwen2Tokenizer'
  tok.encode("嗯，")       -> [106287, 3837]
  tok.decode(...)         -> '嗯，'
  

• Downstream impact: vLLM 0.21.0 pins transformers 5.8.1, so serving this model tokenizes a 14-character Chinese prompt into 3 tokens and streams "ĊåĹ¯" instead of "嗯，".

• This presumably worked under transformers 4.x, where the fast tokenizer loaded tokenizer.json directly and did not follow the declared class here.