BUG: `factorize(use_na_sentinel=True)` ignores `use_na_sentinel` for pre-encoded PyArrow `DictionaryArray`s
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
import pyarrow as pa
for null_encoding in ["mask", "encode"]:
arr = pa.array([*"abc", None]).dictionary_encode(null_encoding=null_encoding)
s = pd.Series(arr, dtype=pd.ArrowDtype(arr.type))
print(
f' {null_encoding = } '.center(80, '='),
s.factorize(use_na_sentinel=True),
sep="\n",
end="\n\n",
)
============================ null_encoding = 'mask' ============================
(array([ 0, 1, 2, -1]), Index(['a', 'b', 'c'],
dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))
=========================== null_encoding = 'encode' ===========================
(array([0, 1, 2, 3]), Index(['a', 'b', 'c', <NA>],
dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))Issue Description
pandas.factorize produces inconsistent results for PyArrow dictionary-encoded
arrays depending on how the input DictionaryArray was is structured.
Specifically, when the array is created with:
.dictionary_encode(null_encoding="mask").dictionary_encode(null_encoding="encode")
calling factorize(use_na_sentinel=True) produces different outputs, even though both arrays represent the same logical data.
The unexpected case arises from production a pyarrow.DictionaryArray via dictionary_encode(null_encoding="encode") then factorizing with pandas.factorize(..., use_na_sentinel=True)`
In this case, factorize behaves as though use_na_sentinel=False: the missing
value is assigned a regular code (3) and <NA> is included in the returned
uniques.
The bug seems to arise from within the factorize implementation for arrow backed arrays in pandas. Where already dictionary-encoded arrays are only re-encoded when use_na_sentinel=False. However, when use_na_sentinel=True (null_encoding="mask"), the existing DictionaryArray is returned unchanged, preserving whatever null encoding it originally had. As a result, a DictionaryArray created with null_encoding="encode" bypasses the requested use_na_sentinel=True behavior and continues to treat null as a regular dictionary value.
Expected Behavior
I would expect the MRE code to produce the following output, where the code -1 is used to refer to the location of Null values in the input provided that use_na_sentinel=True was used in the pandas.factorize call:
============================ null_encoding = 'mask' ============================
(array([ 0, 1, 2, -1]), Index(['a', 'b', 'c'],
dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))
=========================== null_encoding = 'encode' ===========================
(array([ 0, 1, 2, -1]), Index(['a', 'b', 'c'],
dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))Installed Versions
INSTALLED VERSIONS
commit : ea80cd03d44ece976e86d09198dac28ed4f6319d python : 3.14.2 python-bits : 64 OS : Linux OS-release : 6.18.36-1-lts Version : #1 SMP PREEMPT_DYNAMIC Fri, 19 Jun 2026 12:22:16 +0000 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8
pandas : 3.0.0.dev0+4448.gea80cd03d4 numpy : 2.4.6 dateutil : 2.9.0.post0 pip : 25.3 Cython : 3.2.9 sphinx : 9.1.0 IPython : 9.15.0 adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.15.0 bottleneck : 1.6.0 fastparquet : 2026.5.0 fsspec : 2026.6.0 html5lib : 1.1 hypothesis : 6.161.6 gcsfs : 2026.7.0 jinja2 : 3.1.6 lxml.etree : 6.1.1 matplotlib : 3.11.1 numba : 0.66.0 numexpr : 2.14.2 odfpy : None openpyxl : 3.1.5 psycopg2 : 2.9.12 pymysql : 2.2.8 pyarrow : 23.0.1 pyiceberg : 0.11.1 pyreadstat : 1.3.5 pytest : 9.1.1 python-calamine : None pytz : 2026.3.post1 pyxlsb : 1.0.10 s3fs : 2026.6.0 scipy : 1.18.0 sqlalchemy : 2.0.51 tables : 3.11.1 tabulate : 0.10.0 xarray : 2026.7.0 xlrd : 2.0.2 xlsxwriter : 3.2.9 zstandard : 0.25.0 qtpy : None pyqt5 : None
Source: pandas-dev/pandas