da.to_zarr writes corrupt data when specifying chunks or shards
Author: joshua-gouldCreated Sep 4, 2026Updated Sep 8, 2026
Labelsneeds triage
If a user specified chunks or shards arguments to da.to_zarr that do not align with dask chunks, then zarr file is corrupted:
import dask.array as da
import h5py
a = da.arange(100).reshape((10, 10)).rechunk((2, 2))
da.to_zarr(a, "test1.zarr", overwrite=True)
print((a == da.from_zarr("test1.zarr")).all().compute()) # works
da.to_zarr(a, "test2.zarr", chunks=(4, 4), overwrite=True)
print((a == da.from_zarr("test2.zarr")).all().compute()) # corrupted
da.to_zarr(a, "test3.zarr", shards=(4, 4), overwrite=True)
print((a == da.from_zarr("test3.zarr")).all().compute()) # corrupted
a.to_hdf5("test1.h5", 'x')
with h5py.File("test1.h5", 'r') as f:
print((a == f['x']).all().compute()) # works
# hdf5 seems to take care of misaligned chunks
a.to_hdf5("test2.h5", 'x', chunks=(4, 4))
with h5py.File("test2.h5", 'r') as f:
print((a == f['x']).all().compute()) # works
Environment:
- Dask version: latest from github
- Python version: 3.13
- Install method (conda, pip, source): source
Source: dask/dask