#1663·tokenizers

`PreTrainedTokenizerFast` 在带有标记的文本上行为不一致

作者: sven-nm创建于 2024年10月11日更新于 2026年9月4日
标签bug
  • transformers version: 4.45.2
  • Platform: Linux-5.4.0-193-generic-x86_64-with-glibc2.31
  • Python version: 3.12.7
  • Huggingface_hub version: 0.25.2
  • Safetensors version: 0.4.5
  • Accelerate version: not installed
  • Accelerate config: not found
  • PyTorch version (GPU?): not installed (NA)
  • Tensorflow version (GPU?): not installed (NA)
  • Flax version (CPU?/GPU?/TPU?): not installed (NA)
  • Jax version: not installed
  • JaxLib version: not installed
  • Using distributed or parallel set-up in script?:

Who can help?

内容来源: huggingface/tokenizers