#12543·dask

Unable to save dask dataframe containing list column with pandas==3

Author: joshua-gouldCreated Aug 6, 2026Updated Sep 14, 2026
Labelsneeds attentionneeds triage

Describe the issue: The following code works with pandas<3 but fails using pandas==3.

Minimal Complete Verifiable Example:

python
import dask
import pandas as pd
from dask import delayed
import dask.dataframe as dd

import pyarrow as pa

dask.config.set({"dataframe.convert-string": False})


@delayed
def test():
    return pd.DataFrame(data=dict(a=[[1.0, 2.0]]))


meta = dd.utils.make_meta([("a", object)])
df = dd.from_delayed(test(), meta=meta)

to_parquet_args = dict(schema={"a": pa.list_(pa.float64(), 2)})
df.to_parquet('test.parquet', **to_parquet_args)

"python3.13/site-packages/dask/dataframe/dask_expr/_collection.py", line 3327, in to_parquet
    return to_parquet(self, path, **kwargs)
  File "python3.13/site-packages/dask/dataframe/dask_expr/io/parquet.py", line 594, in to_parquet
    i_offset, fmd, metadata_file_exists, extra_write_kwargs = engine.initialize_write(
                                                              ~~~~~~~~~~~~~~~~~~~~~~~^
        df,
        ^^^
    ...<9 lines>...
        **kwargs,
        ^^^^^^^^^
    )
    ^
  File "python3.13/site-packages/dask/dataframe/io/parquet/arrow.py", line 654, in initialize_write
    inferred_schema = pyarrow_schema_dispatch(
                      ~~~~~~~~~~~~~~~~~~~~~~~^
        df._meta_nonempty.set_index(index_cols)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        if index_cols
        ^^^^^^^^^^^^^
        else df._meta_nonempty
        ^^^^^^^^^^^^^^^^^^^^^^
    ).remove_metadata()
    ^
  File "python3.13/site-packages/dask/utils.py", line 782, in __call__
    return meth(arg, *args, **kwargs)
  File "python3.13/site-packages/dask/dataframe/backends.py", line 238, in get_pyarrow_schema_pandas
    return pa.Schema.from_pandas(obj, preserve_index=preserve_index)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/types.pxi", line 3148, in pyarrow.lib.Schema.from_pandas
  File "python3.13/site-packages/pyarrow/pandas_compat.py", line 593, in dataframe_to_types
    type_ = pa.array(c, from_pandas=True).type
            ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/array.pxi", line 408, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 91, in pyarrow.lib._ndarray_to_array
    check_status(NdarrayToArrow(pool, values, mask, from_pandas,
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert <object object at 0x107b460c0> with type object: did not recognize Python value type when inferring an Arrow data type

Environment:

  • Dask version: 2026.7.1
  • Python version: 3.13
  • Operating System: Mac
  • Install method: pip