#286·gpt-oss

Local Harmony tokenizer assigns token ID 200018 to two special tokens

Author: sylvesterkaczmarekCreated Aug 16, 2026Updated Aug 16, 2026

Summary

gpt_oss.tokenizer.get_tokenizer() copies the o200k_base special tokens and then generates every reserved_* token from 200013 through 201087.

o200k_base already assigns ID 200018 to <|endofprompt|>, so the generated <|reserved_200018|> creates a second special-token name for the same ID.

Impact

The tokenizer's special-token mapping is no longer one-to-one. Encoding <|endofprompt|> yields ID 200018, while inverse special-token lookup can resolve that ID to the later reserved alias instead, breaking special-token round-trip expectations and making the local tokenizer inconsistent with a collision-free vocabulary.

The same collision was identified and fixed independently in openai/tiktoken; the local tokenizer implementation in this repository still reproduces the duplicate.

Proposed resolution

When generating Harmony reserved_* entries, skip IDs already occupied by explicit/base special tokens. Add a regression that asserts all special-token IDs are unique and that <|reserved_200018|> is not created while <|endofprompt|> remains mapped to 200018.