#2412·tokenizers

BertWordPieceTokenizer ignores wordpieces_prefix when loading a vocabulary

Author: HaokaiDingCreated Sep 16, 2026Updated Sep 16, 2026

BertWordPieceTokenizer(..., wordpieces_prefix="@@") configures the decoder with @@, but the underlying WordPiece model keeps its default ## prefix. A vocabulary containing custom-prefixed subwords therefore encodes known text as [UNK].

python
from tokenizers import BertWordPieceTokenizer

tokenizer = BertWordPieceTokenizer(
    vocab={"[UNK]": 0, "[CLS]": 1, "[SEP]": 2, "h": 3, "@@ello": 4},
    wordpieces_prefix="@@",
)
output = tokenizer.encode("hello", add_special_tokens=False)
print(output.tokens)           # Actual: ['[UNK]']
print(tokenizer.decode([3, 4])) # 'hello'
assert output.tokens == ["h", "@@ello"]

Expected: encoding and decoding use the same prefix, yielding ['h', '@@ello'] for hello.

This also breaks the training/save/reload workflow: training with wordpieces_prefix="@@" works, but loading the saved vocabulary with BertWordPieceTokenizer.from_file(..., wordpieces_prefix="@@") changes the same input to [UNK].

Both WordPiece(...) constructor branches in bert_wordpiece.py omit continuing_subword_prefix. Forwarding wordpieces_prefix fixes dictionary construction and vocabulary-file reloads while preserving the default ## behavior.

Reproduced with the released tokenizers==0.23.2 package and with the current main Python wrappers loaded over that real native extension. Python 3.13.15, macOS ARM64; no model downloads or Rust rebuild. I checked existing issues and open PRs and did not find the same parameter-forwarding fix.