concatenate two BPE tokenizer
Author: mackmakeCreated Feb 4, 2024Updated Feb 4, 2024
hello. i saw in a paper that they combined two tokenizer, one of them was trained by authors and the second was a pretrained one. is it possible to do this without having their training data? if yes, how should we put merges in a correct way?
Source: karpathy/minGPT