#14010·spaCy

German tokenizer splits gender-inclusive colon forms (Kund:innen)

Author: lsmith77Created Aug 10, 2026Updated Aug 10, 2026

How to reproduce the behaviour

The German infix rules split a colon between letters, which breaks gender-inclusive forms, standard orthography in contemporary German business and public-sector text, before tagging or NER ever see the word:

python
import spacy

nlp = spacy.load("de_core_news_lg")  # tokenizer behavior; any de pipeline
print([t.text for t in nlp("Unsere Kund:innen und Mitarbeiter:innen sind zufrieden.")])
# ['Unsere', 'Kund', ':', 'innen', 'und', 'Mitarbeiter', ':', 'innen', 'sind', 'zufrieden', '.']
print([t.text for t in nlp("Wir suchen eine:n Ärzt:in für unser Team.")])
# ['Wir', 'suchen', 'eine', ':', 'n', 'Ärzt', ':', 'in', 'für', 'unser', 'Team', '.']

Expected: Kund:innen, Mitarbeiter:innen, eine:n, Ärzt:in as single tokens. For comparison, the star and interpunct variants already stay whole (Kolleg*innen, Expert·innen), only the colon splits, via this infix in spacy/lang/de/punctuation.py:

python
r"(?<=[{a}])[:<>=](?=[{a}])".format(a=ALPHA)

The gender colon is the most common separator in German inclusive writing (the INCLUSIFY benchmark, arXiv:2212.02564, documents frequency across sources). Every downstream component degrades on the split: the noun is tagged in pieces, NER never sees the name-like whole, and lemmas are computed for the fragments. We created an inclusive language text checker and patch this infix; we'd like to upstream the behavior.

Two possible fixes, happy to submit a PR for either:

  1. Conservative: exempt the noun gender endings from the colon infix - r"(?<=[{a}]):(?!in(nen)?\b)(?=[{a}])" - which keeps Kund:in and Kund:innen whole. Article forms (eine:n, jede:r) would still split; covering them too means exempting short lowercase tails, e.g. r"(?<=[{a}]):(?![a-zäöüß]{1,5}\b)(?=[{a}])", at the cost of no longer splitting all-lowercase typos like wort:wort.
  2. A tokenizer config option, if changing the default is unwanted.

Your Environment

  • Operating System: macOS-26.6.1-arm64-arm-64bit
  • Python Version Used: 3.12.9
  • spaCy Version Used: 3.8.14
  • Environment Information: de_core_news_lg 3.8.0 (behavior identical on 3.7.x; long-standing default, not a regression)

Notes

  • The original solution was developed without AI assistance; however, this ticket was written with the help of Claude. I was unable to find an AI policy for this project.
  • Our production patch keeps <>= splitting and drops only the colon.
  • I have another ticket in the pipeline related to ADJA mistagging of colon forms, which is only observable once they survive tokenization.