Skip to content

zarr3: read bounds in the array metadata and the offset chunk grid - #296

Draft
rhoadesScholar wants to merge 7 commits into
google:masterfrom
rhoadesScholar:zarr3-offset-chunk-key-encoding
Draft

rhoadesScholar wants to merge 7 commits into
google:masterfrom
rhoadesScholar:zarr3-offset-chunk-key-encoding

Conversation

@rhoadesScholar

@rhoadesScholar rhoadesScholar commented Oct 15, 2026 •

Copy link
Copy Markdown

Prototype for ZEP 12 (zarr-developers/zeps#72, spec text zarr-developers/zarr-specs#400): bounds as an alternative to shape in the array metadata ({"inclusive_min": [...], "exclusive_max": [...]}), negative chunk grid indices on the regular grid, and the offset wrapper chunk grid of zarr-developers/zarr-extensions#81 (offset + base_chunk_grid, a regular base only).

  • Domain and grid. The domain starts at inclusive_min, and the internal grid space is the array coordinate minus the grid origin (the offset grid's offset, else 0). Cell indices are therefore the absolute chunk grid indices, negative where they should be, and they stay put when the lower bound moves.
  • Resize. The lower bound of an array whose metadata gives bounds is implicit, so Resize can grow or shrink it. Chunks that fall off the front are deleted through the existing box-difference path. Arrays with shape are unchanged.
  • Creating. A schema domain whose lower bound is below 0 writes bounds; otherwise shape. The offset grid is written only when the spec asks for it.
  • Keys. Parsing accepts a leading - and rejects -0.
  • Fill value. The per-component fill value is broadcast over an unbounded domain, since chunks may sit at negative internal coordinates.
  • Layout and compatibility. The chunk layout reports the offset as the grid origin. The compatibility key ignores the lower bound's value, as it ignores shape, but does distinguish the shape and bounds forms, so a live handle fails loudly rather than reusing a cache whose transform starts at 0. zarr-python keeps the stored form on update (Add Array.prepend, bounds in the array metadata, and the offset chunk grid zarr-developers/zarr-python#4516), so its resizes of a bounds array written here do not trip that check.
  • Also applies the rank-0 prefix fix to ParseKey from review.

Before and after. Before, Open refuses such an array because shape is missing or the chunk grid is unknown. After:

  • GrowsInFrontByEditingZarrJsonAlone writes a shape array, rewrites only zarr.json to grow it by two elements in front (less than a chunk), and reopens it with domain [-2, 6). It reads the old data at its old coordinates, then writes the front without any existing chunk moving.
  • ResizeMovesTheLowerBound grows the front through tensorstore::Resize, shrinks it again, and checks that exactly the chunks that fell off are deleted.
  • CreatedFromADomainBelowZeroWritesBounds creates from a schema domain [-2, 4) and finds bounds in zarr.json.

Also checked: three stores written by zarr-python (zarr-developers/zarr-python#4516), one grown in front by prepend, one on an offset grid with a non-zero offset on both axes, and the same with a transpose codec, read here with the same domains, grid origin and values. In the other direction, a bounds store and an offset-grid store written here read back in zarr-python with the same domain, grid origin, keys and values, and parse in neuroglancer with the same bounds and grid origin.

Tests. The metadata tests cover both forms and their round trips, shape with bounds and neither rejected, inclusive_min above exclusive_max rejected, origin under regular rejected, a nested or non-regular base under offset rejected, constraints in either form, and the compatibility key. The driver tests cover the domain and grid origin, keys, reopening, growth by a metadata edit, resizing the lower bound both ways, storage statistics over negative keys, and -0. bazel test //tensorstore/driver/zarr3/... passes.

History: the first version of this PR used an offset chunk key encoding, the second a regular_bounded chunk grid; both were replaced on review in zarr-developers/zarr-extensions#79 and zarr-developers/zeps#72.

@google-cla

google-cla Bot commented Oct 15, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for the 'offset' chunk key encoding in the Zarr3 driver, allowing arrays to grow in the negative direction of a dimension by applying a fixed per-dimension integer offset to chunk grid positions. The changes include metadata parsing, validation, key formatting/parsing logic, schema updates, and comprehensive unit tests. The review feedback highlights two issues in driver.cc: first, a potential out-of-bounds or undefined behavior when parsing keys for rank 0 arrays due to incorrect prefix removal length; second, a potential issue in MinGridIndexForLexicographicalOrder where passing a non-positive value to MinValueWithMaxBase10Digits can be avoided by returning -offset directly when the interval consists entirely of negative keys.

Comment thread tensorstore/driver/zarr3/driver.cc Resolved
Comment thread tensorstore/driver/zarr3/driver.cc Resolved
@rhoadesScholar
rhoadesScholar force-pushed the zarr3-offset-chunk-key-encoding branch from e713ac2 to 92e1fec Compare October 15, 2026 23:18
@rhoadesScholar
rhoadesScholar marked this pull request as ready for review October 16, 2026 00:13
@rhoadesScholar

Copy link
Copy Markdown
Author

Ready for review. The encoding is proposed in zarr-developers/zarr-extensions#80; this is the reader side. cc @jbms

@rhoadesScholar
rhoadesScholar marked this pull request as draft October 16, 2026 19:48
@rhoadesScholar

Copy link
Copy Markdown
Author

Converting to draft while the design is worked out in zarr-developers/zarr-extensions#79. This branch is a prototype and will be reworked to match.

@rhoadesScholar

Copy link
Copy Markdown
Author

The metadata form is being settled in ZEP 12 (zarr-developers/zeps#72, spec text zarr-developers/zarr-specs#400). This PR stays on the regular_bounded grid until that lands, then moves its metadata read to the ZEP's fields; the rest of the change is unaffected.

The `offset` encoding stores a chunk under the `default` key of its grid
index plus a fixed per-dimension integer offset, so that an array grows
in the negative direction of a dimension by whole chunks as a
metadata-only change (zarr-extensions chunk-key-encodings/offset).

Adds the kind to `ChunkKeyEncoding` with its `offset` vector (required
for this kind, rejected for the others, length checked against the rank
in ValidateMetadata), applies it in FormatKey and GetChunkStorageKey, and
accepts a leading `-` in ParseKey. MinGridIndexForLexicographicalOrder
splits off every grid index whose key is negative, since negative base-10
components do not sort numerically, and shifts the digit-count rule for
the rest. Schema docs and tests, including a growth that rewrites only
zarr.json and a shrink that deletes through key ranges.
…y encoding

Rework following zarr-extensions#79. The `offset` chunk key encoding is
removed. `regular_bounded` is a regular chunk grid with an `origin` (the
array coordinate where chunk 0 starts) and a `lower_bound` (the
coordinate of the first element).

The array's domain starts at `lower_bound`, and the internal grid space
is the array coordinate minus `origin`, so chunk grid indices are the
absolute ones and may be negative. The lower bound of a
`regular_bounded` array is implicit, so `Resize` can move it; chunks
that fall off the front are deleted through the existing box-difference
path. `regular` arrays are unchanged.

Key parsing accepts a leading `-` and rejects `-0`. The per-component
fill value is broadcast over an unbounded domain, since chunks may now
sit at negative internal coordinates. The chunk layout reports `origin`
as the grid origin, and the compatibility key ignores `lower_bound`, as
it ignores `shape`.
The chunk-grid origin and the lower bound no longer travel together in a
`regular_bounded` grid.  The lower bound is a core member of the array
metadata: exactly one of `shape` and `bounds` is present, with
`bounds = {"inclusive_min": [...], "exclusive_max": [...]}`,
`inclusive_min[i] <= exclusive_max[i]`, and `shape` derived as the
difference.  The `regular` grid is unchanged (`chunk_shape` only; grid
indices by floor division from 0, so they may be negative), and a grid
whose chunk 0 starts elsewhere is the registry wrapper
`{"name": "offset", "configuration": {"offset": [...],
"base_chunk_grid": {"name": "regular", ...}}}`.  Nested `offset` and
any other base are rejected.

`ZarrMetadata` keeps the lower bound and the grid origin as before, each
behind its own flag (`bounded`, `offset_grid`) so that round trips keep
the form the document used.  The driver is unchanged in behaviour: the
domain starts at the lower bound, chunk space is array space minus the
grid origin, key components may be negative, `-0` is rejected.

Compatibility key: the lower bound's value is not part of it, as `shape`
is not, so moving it is a resize.  The form (`shape` vs `bounds`) is,
through an empty `"bounds"` marker: the form fixes whether the domain's
lower bounds are implicit, and the data cache builds every new handle's
transform from its initial metadata, so a form change under a live cache
would otherwise give a fresh open an explicit lower bound of 0 and a
wrong domain.  The `offset` is part of it, as it moves every chunk.

Creating an array writes `bounds` when the constraints gave `bounds` or
the effective domain's lower bound is non-zero along any dimension, else
`shape`; the `offset` grid only when the constraints asked for it.  A
spec with no `shape` constraint now takes the schema domain's explicit
origin instead of pinning the origin to 0, which is what lets a domain
below zero create a bounded array.  A `shape` constraint is read as
`inclusive_min` all zero, so it does not open an array whose lower bound
has moved.
@rhoadesScholar
rhoadesScholar force-pushed the zarr3-offset-chunk-key-encoding branch from 232a4a2 to 0166efc Compare November 11, 2026 00:34
@rhoadesScholar rhoadesScholar changed the title zarr3: support the regular_bounded chunk grid zarr3: read bounds in the array metadata and the offset chunk grid Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant