[BUG] KeyError: '<|endoftext|>'

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

我下载了代码和模型，运行时发生了以下报错，不知道该怎么解决，大佬们棒棒忙 ChatGLM-6B-main_copy$ python3 cli_demo.py Using unk_token, but it is not set yet. Traceback (most recent call last): File "/srv/workspace/chatglm/ChatGLM-6B-main_copy/cli_demo.py", line 7, in tokenizer = AutoTokenizer.from_pretrained("glm-4-9b-chat", trust_remote_code=True, revision="v1.1.0") File "/home/chatglm/.local/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 679, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) File "/home/chatglm/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1804, in from_pretrained return cls._from_pretrained( File "/home/chatglm/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2031, in _from_pretrained added_tokens = tokenizer.sanitize_special_tokens() File "/home/chatglm/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 842, in sanitize_special_tokens return self.add_tokens(self.all_special_tokens_extended, special_tokens=True) File "/home/chatglm/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 985, in add_tokens return self._add_tokens(new_tokens, special_tokens=special_tokens) File "/home/chatglm/.local/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 421, in _add_tokens and self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token) File "/home/chatglm/.local/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 575, in convert_tokens_to_ids return self._convert_token_to_id_with_added_voc(tokens) File "/home/chatglm/.local/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 588, in _convert_token_to_id_with_added_voc return self._convert_token_to_id(token) File "/home/chatglm/.cache/huggingface/modules/transformers_modules/glm-4-9b-chat/tokenization_chatglm.py", line 89, in _convert_token_to_id return self.mergeable_ranks[token] KeyError: '<|endoftext|>'

Expected Behavior

No response

Steps To Reproduce

我下载了代码和模型，运行时发生了以下报错，不知道该怎么解决，大佬们棒棒忙

Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

No response