In encoders, switch to faster string representation than `object` dtype
Author: itamarstCreated Jun 24, 2026Updated Sep 17, 2026
Right now, sklearn.preprocessing._encoders.BaseEncoder._check_X has on its list of tasks "convert list of strings to object dtype". This is bad from a performance perspective, since:
- You now have a pile of Python objects, and end up doing a lot of Python operations.
- Interactions that involve missing data are extra expensive; I'm going to submit a separate issue/PR to speed one code path, but there may be others.
There are two superior alternatives available, that would allow bypassing Python altogether:
- NumPy's StringDType, with custom support for missing data - you can avoid using annoying nans, for example. Would require NumPy 2.0, but possibly a newer version if sklearn hits any bugs that were fixed in subsequent versions.
- Arrow string arrays, with Arrow's built-in support for missing data. Would require PyArrow, or some other Arrow library.
As an alternative to just switching completely internally, it would also be possible to add support for one, the other, or both in addition to object-based dtypes. Then users who have sufficiently new NumPy or PyArrow already installed would benefit from faster runtime, while still preserving backwards compatibility.
Source: scikit-learn/scikit-learn