train() raises ValueError: max() arg is an empty sequence on empty or whitespace-only input
Author: Mefisto04Created Aug 2, 2026Updated Aug 9, 2026
Both BasicTokenizer.train() and RegexTokenizer.train() call max(stats, key=stats.get) without checking whether stats is empty. When there are no byte pairs to merge, training crashes instead of raising a clear error or stopping gracefully.
Steps to reproduce
from minbpe import BasicTokenizer, RegexTokenizer
for cls in (BasicTokenizer, RegexTokenizer):
t = cls()
t.train("", 260) # ValueError: max() arg is an empty sequence
t.train(" ", 260) # same for RegexTokenizerExpected behavior
Raise a clear ValueError("cannot train on empty text") before the merge loop, or stop early when stats is empty (similar to encode's if pair not in self.merges: break).
Actual behavior
ValueError: max() arg is an empty sequenceat basic.py:35 and regex.py:56.
Suggested fix
stats = get_stats(ids)
if not stats:
break # or: raise ValueError("cannot train on empty text")
pair = max(stats, key=stats.get)Environment
- minbpe: main
- Python: 3.11
Notes
Open PR #54 touches this area but there is no tracking issue. Happy to submit a PR with tests if maintainers want this handled.
Source: karpathy/minbpe