Local Harmony tokenizer assigns token ID 200018 to two special tokens
Summary
gpt_oss.tokenizer.get_tokenizer() copies the o200k_base special tokens and then generates every reserved_* token from 200013 through 201087.
o200k_base already assigns ID 200018 to <|endofprompt|>, so the generated <|reserved_200018|> creates a second special-token name for the same ID.
Impact
The tokenizer's special-token mapping is no longer one-to-one. Encoding <|endofprompt|> yields ID 200018, while inverse special-token lookup can resolve that ID to the later reserved alias instead, breaking special-token round-trip expectations and making the local tokenizer inconsistent with a collision-free vocabulary.
The same collision was identified and fixed independently in openai/tiktoken; the local tokenizer implementation in this repository still reproduces the duplicate.
Proposed resolution
When generating Harmony reserved_* entries, skip IDs already occupied by explicit/base special tokens. Add a regression that asserts all special-token IDs are unique and that <|reserved_200018|> is not created while <|endofprompt|> remains mapped to 200018.
Source: openai/gpt-oss