#2249·presidio

StanzaNlpEngine: 德语多词令牌 (im/am/zum) 替换文档文本,并丢弃所有 NER 实体;缩进的文本以 500 个"未找到词"结束

作者: svkaenel创建于 2026年9月12日更新于 2026年9月12日

Describe the bug

With StanzaNlpEngine and the German model, any text containing a German multi-word token (a preposition-article contraction such as im, am, zum, zur, beim, vom, ins, ans) loses every NER entity: /analyze returns HTTP 200 with an empty list where the same sentence with the contraction written out returns the PERSON. On indented text (JSON snippets, tables, lists) the same defect turns into HTTP 500 Did not find word '...' in the list of tokens although it is expected to be found from LemmaContextAwareEnhancer.

Cause (read from presidio_analyzer/nlp_engine/stanza_nlp_engine.py, same on main today):

  1. StanzaNlpEngine.load hardcodes processors="tokenize,pos,lemma,ner"; Stanza force-adds mwt for de ("Language de package default expects mwt, which has been added") and expands imin + dem.
  2. The inlined spacy-stanza tokenizer flattens token.words in StanzaTokenizer.__get_tokens_with_heads, so the expanded words no longer match the text. __get_words_and_spaces raises, and _convert_doc then replaces the whole doc text by " ".join(tokens) (the "Due to multiword token expansion or an alignment issue, the original text has been replaced by space-separated expanded tokens." warning).
  3. The Stanza NER spans carry offsets of the original text. In the replaced doc they hit no token boundary, char_span returns None, and the entity is silently dropped → 200, empty result.
  4. In indented text the replaced text is shorter than the original, so the token list ends before the text does. For a pattern recognizer hit past the last token, LemmaContextAwareEnhancer._find_index_of_match_token raises ValueError("Did not find word ...") → 500. (The loop condition tokens_indices[i] == start or start < tokens_indices[i] + len(token) accepts any start before the end of some token, so it only fires once the hit lies beyond the tokenizer's coverage. That is why the offending word in the message varies between runs.)

This is the presidio-side face of explosion/spacy-stanza#70 (open since 2021). Because Presidio inlines the tokenizer, the fix has to land here.

内容来源: data-privacy-stack/presidio