Skip to content

[DO NOT MERGE] Experimental base for out-of-tree protobuf extensions - #11712

Draft
connortsui20 wants to merge 15 commits into
developfrom
ct/vortex-proto-base
Draft

connortsui20 wants to merge 15 commits into
developfrom
ct/vortex-proto-base

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

spiraldb/vortex-proto (private for now) pins this branch while it tests whether an extension type can live outside this repository. It stores protobuf messages as a vortex.protobuf extension type and queries them with DataFusion, which needs layout, scan, and DataFusion changes that are still in review or have no PR yet.

Do not review or merge this PR. Each change lands through its own PR, and vortex-proto moves to develop once they have.

The branch stacks the heads of the open PRs on develop without changes, then adds the parts that have no PR yet:

When one of these PRs changes, a new commit on top brings the branch up to date, so every pinned commit stays reachable.

Not included yet: mixed DataFusion and Vortex expressions in scans, which #11431, #11667, and #11669 work on, and a function converter registry in vortex-datafusion, so that a crate outside this repository can convert calls nested inside other expressions. Until the first lands, a projection that falls back to DataFusion reads the whole protobuf column, because #11642 narrows reads only through non-nullable structs and not through extension columns.

This PR was prepared with Claude Code.

An extension column was always written by the leaf strategy, so a column whose
storage is a struct became one segment per block and reading one field read
every field. The new `vortex.ext` layout wraps a layout of the storage, so the
table strategy splits an opted-in extension type's storage into per-field
columns. Readers and the planner rewrite `ext_storage(root)` to the storage
root, so projections, filters and pruning reach the storage columns, and they
wrap whole values back into the extension type themselves.

Decomposition is opt-in per extension type through the layout session and the
unstable edition, so existing types such as the spatial types keep their leaf
layout and statistics.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Field access on an extension column converts to a Vortex field read on its storage, so the Vortex scan evaluates it instead of DataFusion. Taken from the extension field access changes of the protobuf prototype (ba36f90), without the protobuf crate.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Most protobuf message fields are nullable structs, and three gaps made them
slow. The planner scan read every field of a nullable struct for a projection
of one field, because the Pack rules skipped nullable structs. The same skip
made a planner read of any field next to a list-layout child fail with "Plan
vortex.plan.list_pack has no pipeline implementation". And the zoned writer
wrote no zone map for any field beneath a nullable struct.

Projections of a nullable struct field now read that field and the struct
validity. Fields beneath a nullable struct now record the zone statistics that
stay correct when the struct is null: bounds, and whether a zone holds no NaN in
place of the NaN count. Their hidden values can only widen a bound, so pruning
stays conservative, and counts, which those values would make wrong, are not
recorded. This changes the zone maps that every writer produces for such fields.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A struct-backed extension written as a leaf makes a field query load every storage field. Let types opt into a structural layout so explicit storage reads can use the existing struct reader and planner without losing the extension dtype.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A projection with any expression that the converter could not convert fell back
to reading whole top-level columns. With a wide struct or protobuf column, one
unsupported function anywhere in the SELECT list read every field. The scan now
evaluates the largest convertible subexpressions that read columns, such as
field access, and DataFusion evaluates the rest over the scan output. Identical
subexpressions are read once, and residual filters use the same split.

The scan evaluates a fallible subexpression only where DataFusion evaluates it
on every row. Nested field access under a null parent returns null, as it
already does when a whole projection converts.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Commit 91dbd9b recorded whether a zone holds no NaN in place of the NaN count
for fields beneath a nullable struct, and added `vortex.all_non_nan` to the
unstable edition for it. That broke the test that the unstable edition permits
only persisted aggregates, and core editions got float zone maps that could
never prune.

Fields beneath a nullable struct now record the NaN count with the bounds. The
hidden values of null struct rows can only add to the count, so a count of zero
still proves that a zone holds no NaN, and the scans use it only that way.
Float fields beneath nullable structs now prune in every edition, and the
unstable edition is as on develop apart from `vortex.ext`.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A repeated protobuf message is a list of structs, and reading one of its fields,
`list_transform(items, x -> x.sku)`, read every element field: the list reader
knew only validity, offsets or everything. The planner scan could not read a
list layout at all ("Plan vortex.plan.list_pack has no pipeline
implementation").

The classic list reader now projects the elements with the lambda body, so it
reads only the element columns that the lambda uses. The planner compiles
`ListPack` into a source that reads the offsets and validity of the selected
lists first, then scans only the elements of selected, valid lists, with the
same element pushdown. Both scans also move strict functions that cannot fail
below the validity of a nullable struct, so the pushdown reaches lists inside
nullable protobuf messages. A lambda that can fail never runs on elements that
in-memory `list_transform` would skip.

The list layout stays out of the unstable edition.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
`ExpressionConverter` gains `split_filtered_projection` and
`no_pushdown_filtered_projection`, so the opener's residual-filter path uses a
converter's overrides instead of always splitting with the default converter.
DataFusion may evaluate an expression that stays with it on some rows only, so
inside such an expression the scan evaluates only subexpressions that cannot
fail. Partial projection pushdown no longer falls back to raw columns when a
lambda appears inside an expression that stays in DataFusion.

Taken from the general changes of 0cb27a3, without the protobuf functions.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The merged #11638 and the prototype both rewrote field reads beneath the
validity mask of a nullable struct. The merge keeps develop's pack planner and
moves its nested-mask field projection into the prototype's mask lifter, which
also lifts statistics and strict functions that cannot fail.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 added the changelog/skip Do not list PR in the changelog label Nov 20, 2026
A field of a nullable struct reaches the reader wrapped in the struct's
mask, so neither scan could prune with its zone map. Both scans now look
through the mask and keep only proofs that stay true when the zone map
also covers values the struct hides, because releases up to 0.88.0 wrote
zone maps over the stored values and no file records whether a zone map
excludes them.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
DataFusion can see a struct where Vortex sees an opaque extension value. Expose the storage explicitly when the Arrow field uses storage export, so the scan can project the requested field without bypassing exporters that transform values.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A single unsupported projection makes the scan read whole struct inputs. Supply provably equivalent field reads from the scan so DataFusion can finish the enclosing expressions without reading unused siblings. Custom converters retain control of the projection split.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A field beneath a nullable struct holds values in rows where the struct is
null, so develop writes no zone map for it, and filters on fields inside
nullable structs never prune. Bounds stay correct for the field as read through
the struct, because the hidden values can only loosen them, and a NaN count of
zero still proves that a zone holds no NaN. Such a field now records its bounds
and its NaN count, and no counts that the hidden rows would make wrong. The
scans from #11710 read these zone maps only in that conservative way.

This is the writer half of the prototype's pruning beneath nullable structs
(91dbd9b and 7ac0dbb). #10961 rewrites the same writer to aggregate only
the visible values.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A repeated protobuf message is a list of structs, and reading one of
its fields, `list_transform(items, x -> x.sku)`, read every element
field: the list reader knew only validity, offsets or everything. The
planner scan could not read a list layout at all ("Plan
vortex.plan.list_pack has no pipeline implementation").

The classic list reader now projects the elements with the lambda
body, so it reads only the element columns that the lambda uses. The
planner compiles `ListPack` into a source that reads the offsets and
validity of the selected lists first, then scans only the elements of
selected, valid lists, with the same element pushdown. Both scans also
move strict functions that cannot fail below the validity of a
nullable struct, so the pushdown reaches lists inside nullable
protobuf messages. A lambda that can fail never runs on elements that
in-memory `list_transform` would skip.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The branch carried the prototype versions of the changes that the open
PRs make, and those PRs have moved on in review. For example, #11640
now names the layout `vortex.extension`. This merge takes the tree of a
stack on develop at 967bcb6, which holds the PR heads and ports of
the two changes that have no PR yet:

- #11640 at 24fd9da.
- #11710 at add4115.
- #11639 at 4f25e2f. Its newer head, 115318b, fails three of its
  own tests.
- #11642 at d16038c.
- Zone maps of bounds beneath nullable structs.
- Single fields of list elements on both scans.

A merge instead of a force-push keeps every commit that the branch
published reachable.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/skip Do not list PR in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant