BertWordPieceTokenizer ignores wordpieces_prefix when loading a vocabulary
BertWordPieceTokenizer(..., wordpieces_prefix="@@") configures the decoder with @@, but the underlying WordPiece model keeps its default ## prefix. A vocabulary containing custom-prefixed subwords therefore encodes known text as [UNK].
from tokenizers import BertWordPieceTokenizer
tokenizer = BertWordPieceTokenizer(
vocab={"[UNK]": 0, "[CLS]": 1, "[SEP]": 2, "h": 3, "@@ello": 4},
wordpieces_prefix="@@",
)
output = tokenizer.encode("hello", add_special_tokens=False)
print(output.tokens) # Actual: ['[UNK]']
print(tokenizer.decode([3, 4])) # 'hello'
assert output.tokens == ["h", "@@ello"]Expected: encoding and decoding use the same prefix, yielding ['h', '@@ello'] for hello.
This also breaks the training/save/reload workflow: training with wordpieces_prefix="@@" works, but loading the saved vocabulary with BertWordPieceTokenizer.from_file(..., wordpieces_prefix="@@") changes the same input to [UNK].
Both WordPiece(...) constructor branches in bert_wordpiece.py omit continuing_subword_prefix. Forwarding wordpieces_prefix fixes dictionary construction and vocabulary-file reloads while preserving the default ## behavior.
Reproduced with the released tokenizers==0.23.2 package and with the current main Python wrappers loaded over that real native extension. Python 3.13.15, macOS ARM64; no model downloads or Rust rebuild. I checked existing issues and open PRs and did not find the same parameter-forwarding fix.
Source: huggingface/tokenizers