[Data] from_pandas with override_num_blocks argument is broken on pandas version 3
What happened + What you expected to happen
Ray Data uses numpy array_split in from_pandas method when you supply an override_num_blocks argument: https://github.com/ray-project/ray/blob/8291a4497913164ef6e21f9d9ef6a719faf65773/python/ray/data/read_api.py#L4599
Internally numpy array_split calls swapaxes, which, when you pass in a dataframe, calls the pandas swapaxes implementation (somewhat magically): https://github.com/numpy/numpy/blob/0532af47d6a815298b7841de00bdbc547104b237/numpy/lib/_shape_base_impl.py#L797
Pandas removed its swapaxes method in version 3 (deprecated since 2.1.0): https://pandas.pydata.org/pandas-docs/version/2.3/reference/api/pandas.DataFrame.swapaxes.html
Now calling array_split with a dataframe fails silently by returning a list of numpy arrays, instead of a list of dataframes.
See https://github.com/numpy/numpy/issues/24889#issuecomment-1895503977 and https://github.com/numpy/numpy/issues/24889#issuecomment-1981503361
For reference it looks like there's some discussion of adding a native pandas split function: https://github.com/pandas-dev/pandas/issues/62719
For now the fix might be to use something like df.iloc to create multiple dataframes instead of np.array_split. Something like:
dfs = [ary.iloc[idx] for idx in np.array_split(np.arange(len(ary)), k)]Versions / Dependencies
Python : 3.13.14 ray: 2.58.0 pandas: 3.0.5 numpy: 2.5.3
Reproduction script
Good:
df = pd.DataFrame({"a": [1, 2, 3, 4]})
ray.data.from_pandas(df)
...
2026-09-15 11:14:01,390 INFO logging_progress.py:174 -- ======= Running Dataset: dataset_034PNa5jOqWXC8p3JxDQjs_0 =======
2026-09-15 11:14:01,391 INFO logging_progress.py:225 -- Total Progress: 0/?
2026-09-15 11:14:01,391 INFO logging_progress.py:227 -- Active & requested resources: 0/0 CPU, 0.0B/0.0B object store
2026-09-15 11:14:01,392 INFO logging_progress.py:192 -- =================================================================
2026-09-15 11:14:01,403 INFO streaming_executor.py:365 -- ✔️ Dataset dataset_034PNa5jOqWXC8p3JxDQjs_0 execution finished in 0.00 secondsBad:
df = pd.DataFrame({"a": [1, 2, 3, 4]})
ray.data.from_pandas(df, override_num_blocks=2)
...
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[12], line 1
----> 1 ray.data.from_pandas(df, override_num_blocks=2)
File ~.zzz/lib/python3.13/site-packages/ray/data/read_api.py:3882, in from_pandas(dfs, override_num_blocks)
3880 context = DataContext.get_current()
3881 if context.enable_tensor_extension_casting:
-> 3882 dfs = [_cast_ndarray_columns_to_tensor_extension(df.copy()) for df in dfs]
3884 return from_pandas_refs([ray.put(df) for df in dfs])
File ~.zzz/lib/python3.13/site-packages/ray/data/util/data_batch_conversion.py:281, in _cast_ndarray_columns_to_tensor_extension(df)
261 from ray.data._internal.tensor_extensions.pandas import (
262 TensorArray,
263 column_needs_tensor_extension,
264 )
266 # Try to convert any ndarray columns to TensorArray columns.
267 # TODO(Clark): Once Pandas supports registering extension types for type
268 # inference on construction, implement as much for NumPy ndarrays and remove
(...) 279 # With duplicate names, ``df[col_name]`` returns a DataFrame
280 # rather than a Series, so we select and assign by position instead.
--> 281 columns_unique = df.columns.is_unique
282 for i, (col_name, dtype) in enumerate(df.dtypes.items()):
283 if (
284 dtype.type is not np.object_
285 ): # Short circuit if non-object type before materializing the column
AttributeError: 'numpy.ndarray' object has no attribute 'columns'(Returned object is now ndarray not dataframe.)
Issue Severity
Medium: It is a significant difficulty but I can work around it.
Source: ray-project/ray