Tokenizer not invertible
Author: t-montesCreated Oct 14, 2024Updated Oct 14, 2024
When encoding `" ..."` through the tokenizer, it's encoded as `" ..."` (one missing space), but tokenizers should be always 100% invertible, aren't they?
Source: bigcode-project/starcoder