#1662·stanza

MWT 令牌 ID 在 to_serialized 后变成列表,导致产生无效的 CoNLL-U

作者: Arthur031221创建于 2026年8月13日更新于 2026年8月14日

Describe the bug

Document.to_serialized() followed by Document.from_serialized() changes a multi-word token id from a tuple to a list, and the document then renders invalid CoNLL-U. The ID column of the range line comes out as the literal text [3, 4] instead of 3-4, and a HEAD of 2 appears on a line that did not have one. The id is a list rather than a tuple afterwards, so using it as a dict key or putting it in a set raises TypeError: unhashable type: 'list', and it no longer compares equal to (3, 4). to_dict() emits an MWT token id as a tuple. JSON has no tuple type, so it comes back as a list, and nothing on the read path converts it back. dict_to_conll_text (doc.py:1154) tests isinstance(token_dict[key], tuple), sees a list, and falls through to str([3, 4]). Because - is then not in that string, the branch at doc.py:1172 that inserts a dummy head also fires. This affects every language with multi-word tokens. To Reproduce No models needed.

内容来源: stanfordnlp/stanza