#66490·pandas

BUG: `factorize(use_na_sentinel=True)` ignores `use_na_sentinel` for pre-encoded PyArrow `DictionaryArray`s

Author: camriddellCreated Jul 27, 2026Updated Sep 28, 2026
LabelsBugAlgosArrow

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

python
import pandas as pd
import pyarrow as pa

for null_encoding in ["mask", "encode"]:
    arr = pa.array([*"abc", None]).dictionary_encode(null_encoding=null_encoding)
    s = pd.Series(arr, dtype=pd.ArrowDtype(arr.type))
    print(
        f' {null_encoding = } '.center(80, '='),
        s.factorize(use_na_sentinel=True),
        sep="\n",
        end="\n\n",
    )



============================ null_encoding = 'mask' ============================
(array([ 0,  1,  2, -1]), Index(['a', 'b', 'c'],
      dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))

=========================== null_encoding = 'encode' ===========================
(array([0, 1, 2, 3]), Index(['a', 'b', 'c', <NA>],
      dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))

Issue Description

pandas.factorize produces inconsistent results for PyArrow dictionary-encoded arrays depending on how the input DictionaryArray was is structured.

Specifically, when the array is created with:

  • .dictionary_encode(null_encoding="mask")
  • .dictionary_encode(null_encoding="encode")

calling factorize(use_na_sentinel=True) produces different outputs, even though both arrays represent the same logical data.

The unexpected case arises from production a pyarrow.DictionaryArray via dictionary_encode(null_encoding="encode") then factorizing with pandas.factorize(..., use_na_sentinel=True)`

In this case, factorize behaves as though use_na_sentinel=False: the missing value is assigned a regular code (3) and <NA> is included in the returned uniques.

The bug seems to arise from within the factorize implementation for arrow backed arrays in pandas. Where already dictionary-encoded arrays are only re-encoded when use_na_sentinel=False. However, when use_na_sentinel=True (null_encoding="mask"), the existing DictionaryArray is returned unchanged, preserving whatever null encoding it originally had. As a result, a DictionaryArray created with null_encoding="encode" bypasses the requested use_na_sentinel=True behavior and continues to treat null as a regular dictionary value.

Expected Behavior

I would expect the MRE code to produce the following output, where the code -1 is used to refer to the location of Null values in the input provided that use_na_sentinel=True was used in the pandas.factorize call:

============================ null_encoding = 'mask' ============================
(array([ 0,  1,  2, -1]), Index(['a', 'b', 'c'],
      dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))

=========================== null_encoding = 'encode' ===========================
(array([ 0,  1,  2, -1]), Index(['a', 'b', 'c'],
      dtype='dictionary<values=string, indices=int32, ordered=0>[pyarrow]'))

Installed Versions

INSTALLED VERSIONS

commit : ea80cd03d44ece976e86d09198dac28ed4f6319d python : 3.14.2 python-bits : 64 OS : Linux OS-release : 6.18.36-1-lts Version : #1 SMP PREEMPT_DYNAMIC Fri, 19 Jun 2026 12:22:16 +0000 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 3.0.0.dev0+4448.gea80cd03d4 numpy : 2.4.6 dateutil : 2.9.0.post0 pip : 25.3 Cython : 3.2.9 sphinx : 9.1.0 IPython : 9.15.0 adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.15.0 bottleneck : 1.6.0 fastparquet : 2026.5.0 fsspec : 2026.6.0 html5lib : 1.1 hypothesis : 6.161.6 gcsfs : 2026.7.0 jinja2 : 3.1.6 lxml.etree : 6.1.1 matplotlib : 3.11.1 numba : 0.66.0 numexpr : 2.14.2 odfpy : None openpyxl : 3.1.5 psycopg2 : 2.9.12 pymysql : 2.2.8 pyarrow : 23.0.1 pyiceberg : 0.11.1 pyreadstat : 1.3.5 pytest : 9.1.1 python-calamine : None pytz : 2026.3.post1 pyxlsb : 1.0.10 s3fs : 2026.6.0 scipy : 1.18.0 sqlalchemy : 2.0.51 tables : 3.11.1 tabulate : 0.10.0 xarray : 2026.7.0 xlrd : 2.0.2 xlsxwriter : 3.2.9 zstandard : 0.25.0 qtpy : None pyqt5 : None