The special_tokens in tokenizer should also be controlled by do_lower_case in encoder_config.

I'm come from the vllm community

I'm fixing #20750

I found that sentence-transformers also have this problem

https://github.com/UKPLab/sentence-transformers/blob/d5c8f5181daa468c63041fde18c659c4e4267e77/sentence_transformers/models/Transformer.py#L490-L492

Bert model special_tokens are uppercase, such as [UNK], but "do_lower_case": true in sentence_bert_config.json

So we need Reverse lower() for special_tokens to make it work properly

from sentence_transformers import SentenceTransformer
model_name = "BAAI/bge-base-en"
prompts = ['[UNK]' * 510]
model = SentenceTransformer(model_name)

tokenizer = model.tokenizer.__call__

def tokenizer_call(self, *args, **kwargs):
    kwargs.pop("truncation", None)
    return tokenizer(*args, **kwargs)

model.tokenizer.__class__.__call__ = tokenizer_call

model.encode(prompts).shape
# RuntimeError: The size of tensor a (2042) must match the size of tensor b (512) at non-singleton dimension 1

I hope vllm and sentence_transformers use exactly the same implementation to fix this problem

So ask if there is any better way to fix it

---

I found that tokenizer.add_special_tokens can reset special_tokens, allowing a more efficient way to fix this.

            special_tokens_map = {
                k: v.lower()
                for k, v in tokenizer.special_tokens_map.items()
            }
            tokenizer.add_special_tokens(special_tokens_map)