MWT token ids become lists after to_serialized, producing invalid CoNLL-U

Describe the bug

Document.to_serialized() followed by Document.from_serialized() changes a multi-word token id from a tuple to a list, and the document then renders invalid CoNLL-U. The ID column of the range line comes out as the literal text [3, 4] instead of 3-4, and a HEAD of 2 appears on a line that did not have one. The id is a list rather than a tuple afterwards, so using it as a dict key or putting it in a set raises TypeError: unhashable type: 'list', and it no longer compares equal to (3, 4).

to_dict() emits an MWT token id as a tuple. JSON has no tuple type, so it comes back as a list, and nothing on the read path converts it back. dict_to_conll_text (doc.py:1154) tests isinstance(token_dict[key], tuple), sees a list, and falls through to str([3, 4]). Because - is then not in that string, the branch at doc.py:1172 that inserts a dummy head also fires.

This affects every language with multi-word tokens.

To Reproduce

No models needed.

from stanza.models.common.doc import Document
from stanza.utils.conll import CoNLL

CONLLU = """# sent_id = 1
# text = Le chien du voisin
1\tLe\tle\tDET\t_\t_\t2\tdet\t_\t_
2\tchien\tchien\tNOUN\t_\t_\t0\troot\t_\t_
3-4\tdu\t_\t_\t_\t_\t_\t_\t_\t_
3\tde\tde\tADP\t_\t_\t5\tcase\t_\t_
4\tle\tle\tDET\t_\t_\t5\tdet\t_\t_
5\tvoisin\tvoisin\tNOUN\t_\t_\t2\tnmod\t_\tSpaceAfter=No

"""

doc = CoNLL.conll2doc(input_str=CONLLU)
back = Document.from_serialized(doc.to_serialized())
print("before:", repr(doc.sentences[0].tokens[2].id))
print("after :", repr(back.sentences[0].tokens[2].id))
print("{:C}".format(back))

before: (3, 4)
after : [3, 4]
# sent_id = 1
# text = Le chien du voisin
1	Le	le	DET	_	_	2	det	_	_
2	chien	chien	NOUN	_	_	0	root	_	_
[3, 4]	du	_	_	_	_	2	_	_	_
3	de	de	ADP	_	_	5	case	_	_
4	le	le	DET	_	_	5	det	_	_
5	voisin	voisin	NOUN	_	_	2	nmod	_	SpaceAfter=No

Expected behavior

The round trip gives back the CoNLL-U it started with, and token.id stays a tuple.

Environment (please complete the following information):

• OS: Ubuntu Linux
• Python version: 3.12.3
• Stanza version: 1.15.0, checked out from dev at 9783c565

Additional context

The to_serialized half of this starts at 031ab2e4, the move from pickle to json. git tag --contains puts that commit in v1.14.0, so it is in the released package as well as on dev. Feeding from_serialized a pickle blob, which it still accepts, gives back 3-4 unchanged. I am not suggesting going back to pickle. The list only needs converting on the way in.

Running the same probe against 031ab2e4^: Document(doc.to_dict()) was correct there and is correct now, but Document(json.loads(json.dumps(doc.to_dict()))) gave [3, 4] there too, so that path was already broken before that commit and is not part of the regression. _process_tokens has no branch for a list id at any commit I looked at.

This is a different defect from #1658 and its PR #1659, which are about FEATS being dropped from MWT tokens. I read the diff on #1659 and it does not touch the id handling, so the two do not overlap, though they are in the same two files.

I found this by reading the serialization code rather than by hitting it in my own work. I have a regression test in stanza/tests/common/test_data_conversion.py that fails before and passes after, and it passes with either of two fixes. Converting the list in the json branch of from_serialized is 8 lines and fixes the round trip only. An elif for a list beside the existing int branch in _process_tokens is 2 lines and fixes the to_dict path above as well. I could not measure a difference between them on a 16000 token document. Which site you want seemed like your call rather than mine. I can open a PR against dev if that is useful.