counting pairs is inaccurate for repeating tokens?
Author: JohannesVodCreated Feb 28, 2024Updated Jun 7, 2024
I just noticed that counting pairs might be slightly inaccurate for a lot of repeating tokens. For example in the sequence 1, 1, 1, 1 the pair (1, 1) gets counted 3 times, but we can only replace it two times. This is slightly inaccurate because for example in the sequence 1, 1, 1, 1, 2, 1, 2, 1, 2 we might choose the pair (1, 1) over (1, 2)
Source: karpathy/minbpe