Repository navigation
[DO NOT MERGE] Experimental base for out-of-tree protobuf extensions - #11712
Draft
connortsui20 wants to merge 15 commits into
Draft
connortsui20 wants to merge 15 commits into
connortsui20 wants to merge 15 commits into
Conversation
An extension column was always written by the leaf strategy, so a column whose storage is a struct became one segment per block and reading one field read every field. The new `vortex.ext` layout wraps a layout of the storage, so the table strategy splits an opted-in extension type's storage into per-field columns. Readers and the planner rewrite `ext_storage(root)` to the storage root, so projections, filters and pruning reach the storage columns, and they wrap whole values back into the extension type themselves. Decomposition is opt-in per extension type through the layout session and the unstable edition, so existing types such as the spatial types keep their leaf layout and statistics. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Field access on an extension column converts to a Vortex field read on its storage, so the Vortex scan evaluates it instead of DataFusion. Taken from the extension field access changes of the protobuf prototype (ba36f90), without the protobuf crate. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Most protobuf message fields are nullable structs, and three gaps made them slow. The planner scan read every field of a nullable struct for a projection of one field, because the Pack rules skipped nullable structs. The same skip made a planner read of any field next to a list-layout child fail with "Plan vortex.plan.list_pack has no pipeline implementation". And the zoned writer wrote no zone map for any field beneath a nullable struct. Projections of a nullable struct field now read that field and the struct validity. Fields beneath a nullable struct now record the zone statistics that stay correct when the struct is null: bounds, and whether a zone holds no NaN in place of the NaN count. Their hidden values can only widen a bound, so pruning stays conservative, and counts, which those values would make wrong, are not recorded. This changes the zone maps that every writer produces for such fields. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A struct-backed extension written as a leaf makes a field query load every storage field. Let types opt into a structural layout so explicit storage reads can use the existing struct reader and planner without losing the extension dtype. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A projection with any expression that the converter could not convert fell back to reading whole top-level columns. With a wide struct or protobuf column, one unsupported function anywhere in the SELECT list read every field. The scan now evaluates the largest convertible subexpressions that read columns, such as field access, and DataFusion evaluates the rest over the scan output. Identical subexpressions are read once, and residual filters use the same split. The scan evaluates a fallible subexpression only where DataFusion evaluates it on every row. Nested field access under a null parent returns null, as it already does when a whole projection converts. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Commit 91dbd9b recorded whether a zone holds no NaN in place of the NaN count for fields beneath a nullable struct, and added `vortex.all_non_nan` to the unstable edition for it. That broke the test that the unstable edition permits only persisted aggregates, and core editions got float zone maps that could never prune. Fields beneath a nullable struct now record the NaN count with the bounds. The hidden values of null struct rows can only add to the count, so a count of zero still proves that a zone holds no NaN, and the scans use it only that way. Float fields beneath nullable structs now prune in every edition, and the unstable edition is as on develop apart from `vortex.ext`. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A repeated protobuf message is a list of structs, and reading one of its fields,
`list_transform(items, x -> x.sku)`, read every element field: the list reader
knew only validity, offsets or everything. The planner scan could not read a
list layout at all ("Plan vortex.plan.list_pack has no pipeline
implementation").
The classic list reader now projects the elements with the lambda body, so it
reads only the element columns that the lambda uses. The planner compiles
`ListPack` into a source that reads the offsets and validity of the selected
lists first, then scans only the elements of selected, valid lists, with the
same element pushdown. Both scans also move strict functions that cannot fail
below the validity of a nullable struct, so the pushdown reaches lists inside
nullable protobuf messages. A lambda that can fail never runs on elements that
in-memory `list_transform` would skip.
The list layout stays out of the unstable edition.
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
`ExpressionConverter` gains `split_filtered_projection` and `no_pushdown_filtered_projection`, so the opener's residual-filter path uses a converter's overrides instead of always splitting with the default converter. DataFusion may evaluate an expression that stays with it on some rows only, so inside such an expression the scan evaluates only subexpressions that cannot fail. Partial projection pushdown no longer falls back to raw columns when a lambda appears inside an expression that stays in DataFusion. Taken from the general changes of 0cb27a3, without the protobuf functions. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The merged #11638 and the prototype both rewrote field reads beneath the validity mask of a nullable struct. The merge keeps develop's pack planner and moves its nested-mask field projection into the prototype's mask lifter, which also lifts statistics and strict functions that cannot fail. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A field of a nullable struct reaches the reader wrapped in the struct's mask, so neither scan could prune with its zone map. Both scans now look through the mask and keep only proofs that stay true when the zone map also covers values the struct hides, because releases up to 0.88.0 wrote zone maps over the stored values and no file records whether a zone map excludes them. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
DataFusion can see a struct where Vortex sees an opaque extension value. Expose the storage explicitly when the Arrow field uses storage export, so the scan can project the requested field without bypassing exporters that transform values. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A single unsupported projection makes the scan read whole struct inputs. Supply provably equivalent field reads from the scan so DataFusion can finish the enclosing expressions without reading unused siblings. Custom converters retain control of the projection split. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A field beneath a nullable struct holds values in rows where the struct is null, so develop writes no zone map for it, and filters on fields inside nullable structs never prune. Bounds stay correct for the field as read through the struct, because the hidden values can only loosen them, and a NaN count of zero still proves that a zone holds no NaN. Such a field now records its bounds and its NaN count, and no counts that the hidden rows would make wrong. The scans from #11710 read these zone maps only in that conservative way. This is the writer half of the prototype's pruning beneath nullable structs (91dbd9b and 7ac0dbb). #10961 rewrites the same writer to aggregate only the visible values. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A repeated protobuf message is a list of structs, and reading one of
its fields, `list_transform(items, x -> x.sku)`, read every element
field: the list reader knew only validity, offsets or everything. The
planner scan could not read a list layout at all ("Plan
vortex.plan.list_pack has no pipeline implementation").
The classic list reader now projects the elements with the lambda
body, so it reads only the element columns that the lambda uses. The
planner compiles `ListPack` into a source that reads the offsets and
validity of the selected lists first, then scans only the elements of
selected, valid lists, with the same element pushdown. Both scans also
move strict functions that cannot fail below the validity of a
nullable struct, so the pushdown reaches lists inside nullable
protobuf messages. A lambda that can fail never runs on elements that
in-memory `list_transform` would skip.
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The branch carried the prototype versions of the changes that the open PRs make, and those PRs have moved on in review. For example, #11640 now names the layout `vortex.extension`. This merge takes the tree of a stack on develop at 967bcb6, which holds the PR heads and ports of the two changes that have no PR yet: - #11640 at 24fd9da. - #11710 at add4115. - #11639 at 4f25e2f. Its newer head, 115318b, fails three of its own tests. - #11642 at d16038c. - Zone maps of bounds beneath nullable structs. - Single fields of list elements on both scans. A merge instead of a force-push keeps every commit that the branch published reachable. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
spiraldb/vortex-proto(private for now) pins this branch while it tests whether an extension type can live outside this repository. It stores protobuf messages as avortex.protobufextension type and queries them with DataFusion, which needs layout, scan, and DataFusion changes that are still in review or have no PR yet.Do not review or merge this PR. Each change lands through its own PR, and vortex-proto moves to
developonce they have.The branch stacks the heads of the open PRs on
developwithout changes, then adds the parts that have no PR yet:4f25e2f59, because its newer head fails three of its own tests.When one of these PRs changes, a new commit on top brings the branch up to date, so every pinned commit stays reachable.
Not included yet: mixed DataFusion and Vortex expressions in scans, which #11431, #11667, and #11669 work on, and a function converter registry in
vortex-datafusion, so that a crate outside this repository can convert calls nested inside other expressions. Until the first lands, a projection that falls back to DataFusion reads the whole protobuf column, because #11642 narrows reads only through non-nullable structs and not through extension columns.This PR was prepared with Claude Code.