Unable to save dask dataframe containing list column with pandas==3
Author: joshua-gouldCreated Aug 6, 2026Updated Sep 14, 2026
Labelsneeds attentionneeds triage
Describe the issue: The following code works with pandas<3 but fails using pandas==3.
Minimal Complete Verifiable Example:
import dask
import pandas as pd
from dask import delayed
import dask.dataframe as dd
import pyarrow as pa
dask.config.set({"dataframe.convert-string": False})
@delayed
def test():
return pd.DataFrame(data=dict(a=[[1.0, 2.0]]))
meta = dd.utils.make_meta([("a", object)])
df = dd.from_delayed(test(), meta=meta)
to_parquet_args = dict(schema={"a": pa.list_(pa.float64(), 2)})
df.to_parquet('test.parquet', **to_parquet_args)
"python3.13/site-packages/dask/dataframe/dask_expr/_collection.py", line 3327, in to_parquet
return to_parquet(self, path, **kwargs)
File "python3.13/site-packages/dask/dataframe/dask_expr/io/parquet.py", line 594, in to_parquet
i_offset, fmd, metadata_file_exists, extra_write_kwargs = engine.initialize_write(
~~~~~~~~~~~~~~~~~~~~~~~^
df,
^^^
...<9 lines>...
**kwargs,
^^^^^^^^^
)
^
File "python3.13/site-packages/dask/dataframe/io/parquet/arrow.py", line 654, in initialize_write
inferred_schema = pyarrow_schema_dispatch(
~~~~~~~~~~~~~~~~~~~~~~~^
df._meta_nonempty.set_index(index_cols)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
if index_cols
^^^^^^^^^^^^^
else df._meta_nonempty
^^^^^^^^^^^^^^^^^^^^^^
).remove_metadata()
^
File "python3.13/site-packages/dask/utils.py", line 782, in __call__
return meth(arg, *args, **kwargs)
File "python3.13/site-packages/dask/dataframe/backends.py", line 238, in get_pyarrow_schema_pandas
return pa.Schema.from_pandas(obj, preserve_index=preserve_index)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/types.pxi", line 3148, in pyarrow.lib.Schema.from_pandas
File "python3.13/site-packages/pyarrow/pandas_compat.py", line 593, in dataframe_to_types
type_ = pa.array(c, from_pandas=True).type
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/array.pxi", line 408, in pyarrow.lib.array
File "pyarrow/array.pxi", line 91, in pyarrow.lib._ndarray_to_array
check_status(NdarrayToArrow(pool, values, mask, from_pandas,
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert <object object at 0x107b460c0> with type object: did not recognize Python value type when inferring an Arrow data typeEnvironment:
- Dask version: 2026.7.1
- Python version: 3.13
- Operating System: Mac
- Install method: pip
Source: dask/dask