Inconsistent chunking between `xr.open_zarr` and `xr.open_dataset(..., engine='zarr')` with `chunks="auto"`

[norlandrhagen]

What happened?

Hi there 👋

I was chatting with @jsignell today about chunk sizes and we came across some potentially inconsistent chunking behavior between xr.open_zarr vs xr.open_dataset(..., engine='zarr'), which I assumed would be identical.

# xr.__version__ : '2025.9.1'
# zarr.__version__ : '3.1.3'
import xarray as xr 

ds = xr.tutorial.open_dataset('air_temperature', chunks={})
ds_rechunked = ds.chunk({'time':100,'lat':25, 'lon':53})
ds_rechunked.to_zarr('air_temperature.zarr', consolidated=False, zarr_format=3)

ds1 = xr.open_zarr('air_temperature.zarr', consolidated=False,chunks="auto")
ds1.chunks

# Frozen({'time': (100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, # 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 20), 'lat': (25,), 'lon': (53,)})

ds2 = xr.open_dataset('air_temperature.zarr', consolidated=False,chunks="auto")
ds2.chunks

# Frozen({'time': (2920,), 'lat': (25,), 'lon': (53,)})


from xarray.testing import assert_chunks_equal

assert_chunks_equal(ds1, ds2)

# AssertionError: 

What did you expect to happen?

The same chunking behavior between xr.open_zarr(...) vs xr.open_dataset(...,engine='zarr')

[jsignell]

I looked at the code and it is pretty obvious why this is happening:

xarray/xarray/backends/zarr.py

Lines 1561 to 1567 in f252119

	if chunks == "auto":
	try:
	guess_chunkmanager(
	chunked_array_type
	) # attempt to import that parallel backend
	chunks = {}

I think if we just delete that one line:

xarray/xarray/backends/zarr.py

Line 1567 in f252119

chunks = {}

the behavior will match. But that'll change the chunksize by default for this function, so I'm pretty wary of deleting that line!

Ideally I think there would be a version of "auto" that never splits any chunks. If that existed I think it would be the ideal way of opening a Zarr and xarray wouldn't have to worry about:

xarray/xarray/namedarray/utils.py

Lines 247 to 263 in 00d18bf

	# Warn where requested chunks break preferred chunks, provided that the variable
	# contains data.
	if data.size: # type: ignore[unused-ignore,attr-defined] # DuckArray protocol doesn't include 'size' - should it?
	for dim, size, chunk_sizes in zip(dims, shape, chunk_shape, strict=True):
	if preferred_chunk_sizes := preferred_chunks.get(dim):
	disagreement = _get_breaks_cached(
	size=size,
	chunk_sizes=chunk_sizes,
	preferred_chunk_sizes=preferred_chunk_sizes,
	)
	if disagreement:
	emit_user_level_warning(
	"The specified chunks separate the stored chunks along "
	f'dimension "{dim}" starting at index {disagreement}. This could '
	"degrade performance. Instead, consider rechunking after loading.",
	)

[dcherian]

Ideally I think there would be a version of "auto" that never splits any chunks

I think this is partly why Xarray normalizes "auto"-chunks.

I agree with deleting that line ;)

[keewis]

see also #7495 (comment)

[jsignell]

Yes exactly @keewis! Do you still think that the special handling for "auto" that exists in open_zarr should be pushed down into open_dataset or are you good with the idea of just deleting the chunks = {} override?

[keewis]

I personally don't even use open_zarr, so removing the only difference between the two sounds like a win to me.

However, what I meant two years ago was that open_zarr would default to

xr.open_zarr(path, chunks="native")  # or similar
# open_dataset translates that to `chunks = {}`

which would then be different from

xr.open_zarr(path, chunks="auto")  # `chunks="auto"` is handled by the chunk manager

Not sure if introducing a new string alias is a good idea, but given that dict default values are frowned upon (and the standard None is already taken), it might still be helpful. That is, until frozendict arrives in python, should PEP814 be accepted.