`PreTrainedTokenizerFast` 在带有标记的文本上行为不一致
作者: sven-nm创建于 2024年10月11日更新于 2026年9月4日
标签bug
transformersversion: 4.45.2- Platform: Linux-5.4.0-193-generic-x86_64-with-glibc2.31
- Python version: 3.12.7
- Huggingface_hub version: 0.25.2
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:
Who can help?
内容来源: huggingface/tokenizers