German tokenizer splits gender-inclusive colon forms (Kund:innen)
How to reproduce the behaviour
The German infix rules split a colon between letters, which breaks gender-inclusive forms, standard orthography in contemporary German business and public-sector text, before tagging or NER ever see the word:
import spacy
nlp = spacy.load("de_core_news_lg") # tokenizer behavior; any de pipeline
print([t.text for t in nlp("Unsere Kund:innen und Mitarbeiter:innen sind zufrieden.")])
# ['Unsere', 'Kund', ':', 'innen', 'und', 'Mitarbeiter', ':', 'innen', 'sind', 'zufrieden', '.']
print([t.text for t in nlp("Wir suchen eine:n Ärzt:in für unser Team.")])
# ['Wir', 'suchen', 'eine', ':', 'n', 'Ärzt', ':', 'in', 'für', 'unser', 'Team', '.']Expected: Kund:innen, Mitarbeiter:innen, eine:n, Ärzt:in as single
tokens. For comparison, the star and interpunct variants already stay
whole (Kolleg*innen, Expert·innen), only the colon splits, via this
infix in spacy/lang/de/punctuation.py:
r"(?<=[{a}])[:<>=](?=[{a}])".format(a=ALPHA)The gender colon is the most common separator in German inclusive writing (the INCLUSIFY benchmark, arXiv:2212.02564, documents frequency across sources). Every downstream component degrades on the split: the noun is tagged in pieces, NER never sees the name-like whole, and lemmas are computed for the fragments. We created an inclusive language text checker and patch this infix; we'd like to upstream the behavior.
Two possible fixes, happy to submit a PR for either:
- Conservative: exempt the noun gender endings from the colon infix -
r"(?<=[{a}]):(?!in(nen)?\b)(?=[{a}])"- which keepsKund:inandKund:innenwhole. Article forms (eine:n,jede:r) would still split; covering them too means exempting short lowercase tails, e.g.r"(?<=[{a}]):(?![a-zäöüß]{1,5}\b)(?=[{a}])", at the cost of no longer splitting all-lowercase typos likewort:wort. - A tokenizer config option, if changing the default is unwanted.
Your Environment
- Operating System: macOS-26.6.1-arm64-arm-64bit
- Python Version Used: 3.12.9
- spaCy Version Used: 3.8.14
- Environment Information: de_core_news_lg 3.8.0 (behavior identical on 3.7.x; long-standing default, not a regression)
Notes
- The original solution was developed without AI assistance; however, this ticket was written with the help of Claude. I was unable to find an AI policy for this project.
- Our production patch keeps
<>=splitting and drops only the colon. - I have another ticket in the pipeline related to ADJA mistagging of colon forms, which is only observable once they survive tokenization.
Source: explosion/spaCy