#403·open_clip

Improve tokenizer decode

Author: vturrisiCreated Feb 2, 2023Updated Feb 7, 2023

Right now the tokenizer decode method supports only a single instance at a time. I think it would be good to have `batch_decode` function and also support `skip_special_tokens` and `clean_up_tokenization_spaces` as in huggingface.