#46785·transformers

Is transformers 5.12.1 requires tokenizers<=0.23.0,>=0.22.0 really right?

Author: jrp2014Created Jun 20, 2026Updated Sep 17, 2026

https://github.com/huggingface/tokenizers/releases/tag/v0.23.1 says that:

tokenizers 0.23.1 is the first proper stable release in the 0.23 line — 0.23.0 only ever shipped as rc0 because the release pipeline itself was broken (Node side hadn't shipped multi-platform binaries since 2023, Python side was on pyo3 0.27 without free-threaded support). 0.23.1 is the version where everything actually goes out the door together: full Node multi-platform wheels for the first time in years, Python 3.14 (regular and free-threaded 3.14t), full type hints for every Python class, and a stack of measurable perf wins on the BPE / added-vocab hot paths.

There is no functional 0.23.0 published — we tag 0.23.1 directly so users don't accidentally pull a never-shipped version.

So we should have an upper bound of <0.23 or <0.24?

Source: huggingface/transformers