encode(..., allowed_special="none_raise") raises AssertionError instead of ValueError

Author: Mefisto04Created Aug 2, 2026Updated Aug 2, 2026

RegexTokenizer.encode() documents allowed_special="none_raise" as raising an error when special tokens appear in input. The implementation uses assert:

python

# minbpe/regex.py:137-139
elif allowed_special == "none_raise":
    special = {}
    assert all(token not in text for token in self.special_tokens)

Problems:

Raises AssertionError instead of ValueError
Can be disabled when Python runs with -O

Steps to reproduce

python

from minbpe import RegexTokenizer
from minbpe.gpt4 import GPT4_SPECIAL_TOKENS

t = RegexTokenizer()
t.register_special_tokens(GPT4_SPECIAL_TOKENS)
t.encode("hello <|endoftext|> world", allowed_special="none_raise")
# AssertionError (expected ValueError)

Expected behavior

Raise ValueError naming the disallowed special token.

Suggested fix

python

for token in self.special_tokens:
    if token in text:
        raise ValueError(f"disallowed special token found in text: {token!r}")

Environment

minbpe: main
Python: 3.11

Source: karpathy/minbpe

View original on GitHub View discussion on GitHub