Baike.dev
Log in
Back to tool/Back to issues
#12580·dask

set_index on read_parquet(dtype_backend="pyarrow") raises TypeError: data type '...' not understood for nested Arrow dtypes

Author: guozhansCreated Sep 1, 2026Updated Sep 2, 2026
Labelsneeds triage

Describe the issue

Calling set_index (or anything that shuffles) on a DataFrame created by dd.read_parquet(..., dtype_backend="pyarrow") raises

TypeError: data type 'list<item: struct<type: string, ref: int64, role: string>>[pyarrow]' not understood

when the data contains a nested Arrow dtype (list<...>, list<struct<...>>, map<...>). The same DataFrame computes fine without set_index — including map_partitions doing real work — so the plain read path handles the dtypes correctly and only the shuffle-triggered partition read fails.

The error surfaces from the parquet partition read inside the shuffle graph (read_parquet_part → _arrow_table_to_pandas → Table.to_pandas → pyarrow.pandas_compat._get_extension_dtypes → pandas.pandas_dtype(<dtype str>)), i.e. that read reconstructs dtypes from the parquet pandas metadata — where pa.Table.from_pandas stored the dtype as str(dtype) — instead of using the types_mapper/dtype_backend that the top-level read_parquet call was given. pandas.pandas_dtype() cannot parse nested ArrowDtype string reprs (pandas-dev/pandas#57411, fix in flight in pandas-dev/pandas#66758), so the fallback raises. Flat dtypes like int64[pyarrow] parse fine, which is why the bug only bites nested types.

Even once pandas can parse the strings, the underlying asymmetry — the shuffle-path partition read not receiving the same to_pandas kwargs as the plain read path — seems worth fixing in dask itself.

Minimal Complete Verifiable Example

python
import tempfile
import pandas as pd
import pyarrow as pa
import dask.dataframe as dd

member_type = pa.list_(pa.struct([("type", pa.string()), ("ref", pa.int64()), ("role", pa.string())]))
pdf = pd.DataFrame({
    "id": pd.array([1, 1, 2, 2, 2, 3] * 200, dtype="int64[pyarrow]"),
    "members": pd.array(
        [[{"type": "way", "ref": 10, "role": "outer"}], None,
         [{"type": "node", "ref": 12, "role": ""}]] * 400,
        dtype=pd.ArrowDtype(member_type)),
})

tmp = tempfile.mkdtemp()
dd.from_pandas(pdf, npartitions=4).to_parquet(tmp, write_metadata_file=False)

ddf = dd.read_parquet(tmp, dtype_backend="pyarrow")

ddf.compute()                      # OK
ddf.map_partitions(len).compute()  # OK
ddf.set_index("id").compute()      # TypeError: data type 'list<item: struct<...>>[pyarrow]' not understood

Also fails identically when the frame comes from dd.from_pandas and is written by another process — the trigger is the parquet pandas-metadata round-trip, not the writer.

Traceback

File ".../dask/dataframe/io/parquet/core.py", line 82, in __call__
    return read_parquet_part(
File ".../dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part
    func(
File ".../dask/dataframe/io/parquet/arrow.py", line 590, in read_partition
    df = cls._arrow_table_to_pandas(
File ".../dask/dataframe/io/parquet/arrow.py", line 1805, in _arrow_table_to_pandas
    res = arrow_table.to_pandas(categories=categories, **_kwargs)
File "pyarrow/array.pxi", line 1046, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 5178, in pyarrow.lib.Table._to_pandas
File ".../pyarrow/pandas_compat.py", line 810, in table_to_dataframe
    ext_columns_dtypes = _get_extension_dtypes(
File ".../pyarrow/pandas_compat.py", line 918, in _get_extension_dtypes
    pandas_dtype = _pandas_api.pandas_dtype(dtype)
File "pyarrow/pandas-shim.pxi", line 152, in pyarrow.lib._PandasAPIShim.pandas_dtype
File ".../pandas/core/dtypes/common.py", line 1889, in pandas_dtype
    raise TypeError(f"data type '{dtype}' not understood") from err
TypeError: data type 'list<item: struct<type: string, ref: int64, role: string>>[pyarrow]' not understood

Anything else we need to know?

  • Reproduced on dask 2026.3.0 and 2026.8.0; pandas 3.0.3 and 3.0.5; pyarrow 23.0.1.
  • The one-liner that shows the pandas-side parse failure: pd.api.types.pandas_dtype("list<item: int64>[pyarrow]") → same TypeError (flat "int64[pyarrow]" works). Tracked as pandas-dev/pandas#57411 / #66758.

Environment

  • Dask version: 2026.8.0 (also 2026.3.0)
  • pandas: 3.0.5 (also 3.0.3), pyarrow: 23.0.1
  • Python version: 3.12
  • Operating System: macOS / Linux (both)
  • Install method: pip (uv)

Source: dask/dask

View original on GitHubView discussion on GitHub