Repository navigation
zarr3: read bounds in the array metadata and the offset chunk grid - #296
rhoadesScholar wants to merge 7 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request introduces support for the 'offset' chunk key encoding in the Zarr3 driver, allowing arrays to grow in the negative direction of a dimension by applying a fixed per-dimension integer offset to chunk grid positions. The changes include metadata parsing, validation, key formatting/parsing logic, schema updates, and comprehensive unit tests. The review feedback highlights two issues in driver.cc: first, a potential out-of-bounds or undefined behavior when parsing keys for rank 0 arrays due to incorrect prefix removal length; second, a potential issue in MinGridIndexForLexicographicalOrder where passing a non-positive value to MinValueWithMaxBase10Digits can be avoided by returning -offset directly when the interval consists entirely of negative keys.
e713ac2 to
92e1fec
Compare
|
Ready for review. The encoding is proposed in zarr-developers/zarr-extensions#80; this is the reader side. cc @jbms |
|
Converting to draft while the design is worked out in zarr-developers/zarr-extensions#79. This branch is a prototype and will be reworked to match. |
5ba7353 to
232a4a2
Compare
|
The metadata form is being settled in ZEP 12 (zarr-developers/zeps#72, spec text zarr-developers/zarr-specs#400). This PR stays on the |
The `offset` encoding stores a chunk under the `default` key of its grid index plus a fixed per-dimension integer offset, so that an array grows in the negative direction of a dimension by whole chunks as a metadata-only change (zarr-extensions chunk-key-encodings/offset). Adds the kind to `ChunkKeyEncoding` with its `offset` vector (required for this kind, rejected for the others, length checked against the rank in ValidateMetadata), applies it in FormatKey and GetChunkStorageKey, and accepts a leading `-` in ParseKey. MinGridIndexForLexicographicalOrder splits off every grid index whose key is negative, since negative base-10 components do not sort numerically, and shifts the digit-count rule for the rest. Schema docs and tests, including a growth that rewrites only zarr.json and a shrink that deletes through key ranges.
…y encoding Rework following zarr-extensions#79. The `offset` chunk key encoding is removed. `regular_bounded` is a regular chunk grid with an `origin` (the array coordinate where chunk 0 starts) and a `lower_bound` (the coordinate of the first element). The array's domain starts at `lower_bound`, and the internal grid space is the array coordinate minus `origin`, so chunk grid indices are the absolute ones and may be negative. The lower bound of a `regular_bounded` array is implicit, so `Resize` can move it; chunks that fall off the front are deleted through the existing box-difference path. `regular` arrays are unchanged. Key parsing accepts a leading `-` and rejects `-0`. The per-component fill value is broadcast over an unbounded domain, since chunks may now sit at negative internal coordinates. The chunk layout reports `origin` as the grid origin, and the compatibility key ignores `lower_bound`, as it ignores `shape`.
The chunk-grid origin and the lower bound no longer travel together in a
`regular_bounded` grid. The lower bound is a core member of the array
metadata: exactly one of `shape` and `bounds` is present, with
`bounds = {"inclusive_min": [...], "exclusive_max": [...]}`,
`inclusive_min[i] <= exclusive_max[i]`, and `shape` derived as the
difference. The `regular` grid is unchanged (`chunk_shape` only; grid
indices by floor division from 0, so they may be negative), and a grid
whose chunk 0 starts elsewhere is the registry wrapper
`{"name": "offset", "configuration": {"offset": [...],
"base_chunk_grid": {"name": "regular", ...}}}`. Nested `offset` and
any other base are rejected.
`ZarrMetadata` keeps the lower bound and the grid origin as before, each
behind its own flag (`bounded`, `offset_grid`) so that round trips keep
the form the document used. The driver is unchanged in behaviour: the
domain starts at the lower bound, chunk space is array space minus the
grid origin, key components may be negative, `-0` is rejected.
Compatibility key: the lower bound's value is not part of it, as `shape`
is not, so moving it is a resize. The form (`shape` vs `bounds`) is,
through an empty `"bounds"` marker: the form fixes whether the domain's
lower bounds are implicit, and the data cache builds every new handle's
transform from its initial metadata, so a form change under a live cache
would otherwise give a fresh open an explicit lower bound of 0 and a
wrong domain. The `offset` is part of it, as it moves every chunk.
Creating an array writes `bounds` when the constraints gave `bounds` or
the effective domain's lower bound is non-zero along any dimension, else
`shape`; the `offset` grid only when the constraints asked for it. A
spec with no `shape` constraint now takes the schema domain's explicit
origin instead of pinning the origin to 0, which is what lets a domain
below zero create a bounded array. A `shape` constraint is read as
`inclusive_min` all zero, so it does not open an array whose lower bound
has moved.
232a4a2 to
0166efc
Compare
Prototype for ZEP 12 (zarr-developers/zeps#72, spec text zarr-developers/zarr-specs#400):
boundsas an alternative toshapein the array metadata ({"inclusive_min": [...], "exclusive_max": [...]}), negative chunk grid indices on theregulargrid, and theoffsetwrapper chunk grid of zarr-developers/zarr-extensions#81 (offset+base_chunk_grid, aregularbase only).inclusive_min, and the internal grid space is the array coordinate minus the grid origin (theoffsetgrid'soffset, else 0). Cell indices are therefore the absolute chunk grid indices, negative where they should be, and they stay put when the lower bound moves.boundsis implicit, soResizecan grow or shrink it. Chunks that fall off the front are deleted through the existing box-difference path. Arrays withshapeare unchanged.bounds; otherwiseshape. Theoffsetgrid is written only when the spec asks for it.-and rejects-0.shape, but does distinguish theshapeandboundsforms, so a live handle fails loudly rather than reusing a cache whose transform starts at 0. zarr-python keeps the stored form on update (Add Array.prepend, bounds in the array metadata, and the offset chunk grid zarr-developers/zarr-python#4516), so its resizes of aboundsarray written here do not trip that check.ParseKeyfrom review.Before and after. Before,
Openrefuses such an array becauseshapeis missing or the chunk grid is unknown. After:GrowsInFrontByEditingZarrJsonAlonewrites ashapearray, rewrites onlyzarr.jsonto grow it by two elements in front (less than a chunk), and reopens it with domain[-2, 6). It reads the old data at its old coordinates, then writes the front without any existing chunk moving.ResizeMovesTheLowerBoundgrows the front throughtensorstore::Resize, shrinks it again, and checks that exactly the chunks that fell off are deleted.CreatedFromADomainBelowZeroWritesBoundscreates from a schema domain[-2, 4)and findsboundsinzarr.json.Also checked: three stores written by zarr-python (zarr-developers/zarr-python#4516), one grown in front by
prepend, one on anoffsetgrid with a non-zero offset on both axes, and the same with atransposecodec, read here with the same domains, grid origin and values. In the other direction, aboundsstore and anoffset-grid store written here read back in zarr-python with the same domain, grid origin, keys and values, and parse in neuroglancer with the same bounds and grid origin.Tests. The metadata tests cover both forms and their round trips,
shapewithboundsand neither rejected,inclusive_minaboveexclusive_maxrejected,originunderregularrejected, a nested or non-regular base underoffsetrejected, constraints in either form, and the compatibility key. The driver tests cover the domain and grid origin, keys, reopening, growth by a metadata edit, resizing the lower bound both ways, storage statistics over negative keys, and-0.bazel test //tensorstore/driver/zarr3/...passes.History: the first version of this PR used an
offsetchunk key encoding, the second aregular_boundedchunk grid; both were replaced on review in zarr-developers/zarr-extensions#79 and zarr-developers/zeps#72.