Improve tokenizer decode
Author: vturrisiCreated Feb 2, 2023Updated Feb 7, 2023
Right now the tokenizer decode method supports only a single instance at a time. I think it would be good to have `batch_decode` function and also support `skip_special_tokens` and `clean_up_tokenization_spaces` as in huggingface.
Source: mlfoundations/open_clip