English tokenizer regression in 1.13.0: sentence boundaries inserted mid-sentence around numerals in proper-noun phrases
Describe the bug Upgrading from 1.12.2 to 1.13.0, the default English tokenizer began splitting certain sentences into two at positions with no sentence-final punctuation. The pattern involves a numeral (Arabic or Roman) inside a proper-noun phrase, followed by a capitalized word - the boundary is inserted adjacent to the numeral.
To Reproduce
import stanza
stanza.download(lang="en", processors="tokenize")
nlp = stanza.Pipeline(lang="en", processors="tokenize")
tests = [
"Pope Leo X used this practice to raise funds for rebuilding the grand Saint Peter's Basilica in the Vatican City.",
"Luther's 95 Theses were groundbreaking for several reasons.",
"In conclusion, Martin Luther’s 95 Theses remain a crucial turning point in religious history.",
]
for t in tests:
doc = nlp(t)
print(len(doc.sentences), [s.text for s in doc.sentences])Actual output (1.13.0) - every input splits into 2 sentences:
2 ['Pope Leo', "X used this practice to raise funds for rebuilding the grand Saint Peter's Basilica in the Vatican City."]
2 ["Luther's 95", 'Theses were groundbreaking for several reasons.']
2 ['In conclusion, Martin Luther’s 95', 'Theses remain a crucial turning point in religious history.']Expected behavior One sentence each - there is no sentence-final punctuation at the split points. Versions 1.10.1, 1.11.0, 1.11.1, 1.12.0, 1.12.1, and 1.12.2 all return 1 sentence for all three inputs with their respective default English models (verified individually; models freshly downloaded per version).
Additional context - pattern boundaries The split seems to require a capitalized word after the numeral. These near-identical inputs do not split on 1.13.0:
1 ['Henry VIII had six wives.']
1 ['See Chapter 7 for details of the route.']
1 ['Apollo 11 landed on the moon in 1969.']i.e. "Pope Leo X used" splits, "Henry VIII had" doesn't - suggesting the boundary decision is driven by [numeral][Capitalized word] resembling a sentence start.
Environment
- stanza 1.13.0, default
enpackage, models downloaded fresh viastanza.download - Python 3.11.15, torch 2.7.0, macOS arm64 (CPU)
- Regression bisected: last good release is 1.12.2
Source: stanfordnlp/stanza