#8381·datasets

IterableDataset.to_dict() and to_polars() return an empty generator instead of the data

Author: ebarkhordarCreated Aug 3, 2026Updated Sep 15, 2026

Describe the bug

IterableDataset.to_dict() and IterableDataset.to_polars() each have a bare yield in their batched=True branch, which makes the whole method a generator function. Calling either one the way its docstring shows returns a generator that yields nothing, and the value the else branch returns is swallowed into StopIteration.value. Both are annotated -> Union[dict, Iterator[dict]] / -> Union["pl.DataFrame", Iterator["pl.DataFrame"]], and the plain call is the only example in each docstring.

The sibling to_pandas() had the same shape and was changed to a returned generator expression in #8068; to_dict and to_polars were left as they were. to_list() is unaffected.

Steps to reproduce the bug

from datasets import Dataset

ds = Dataset.from_dict({"col": [0, 1, 2]}).to_iterable_dataset()

print(type(ds.to_dict()))   # <class 'generator'>
print(list(ds.to_dict()))   # []
ds.to_dict()["col"]         # TypeError: 'generator' object is not subscriptable

The data is only reachable through the exception:

g = ds.to_dict()
try:
    next(g)
except StopIteration as e:
    print(e.value)          # {'col': [0, 1, 2]}

ds.to_polars() behaves identically. On the same object, ds.to_pandas() returns a DataFrame and ds.to_list() returns the rows.

Expected behavior

ds.to_dict() returns {'col': [0, 1, 2]} and ds.to_polars() returns a polars.DataFrame, matching Dataset.to_dict() / Dataset.to_polars() and the non-iterator half of each return annotation. batched=True should keep yielding one batch at a time.

Environment info

  • datasets 5.0.2.dev0, main at b7cb10b0e
  • python 3.11.15, Linux
  • pyarrow 25.0.0, polars 1.43.2, pandas 3.0.5