Skip to content

perf(layout): read a path below a nullable struct's field without the rest of the field - #10952

Closed
gitRasheed wants to merge 1 commit into
vortex-data:developfrom
gitRasheed:struct-path-below-nullable-field
Closed

gitRasheed wants to merge 1 commit into
vortex-data:developfrom
gitRasheed:struct-path-below-nullable-field

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Reading a field two levels into nullable structs read the whole inner struct. For t: {x: {p, q}, r} with t and x nullable, projecting $.t.x.q or filtering on $.t.x.q > n read 6.05 MB of a 23.7 MB file, as much as reading all of x. With this change it reads 160 KB. One level, $.s.b, was already pruned.

expand_struct_root rewrites a field access $.x on a nullable struct to mask($.x, is_not_null($)), so the field keeps the struct's nulls. For $.x.q that gives get_item(q, mask($.x, ...)), and the mask hides the path from the partitioner: x's reader gets $ and reads every field. A Spark connector that pushes nested schema pruning down runs into this for any struct inside a nullable struct, since Spark structs are nearly always nullable.

Changes

A get_item/select chain that starts at $.field is rewritten as a whole to mask(chain_on_field, is_not_null($)), so the field's reader gets the path and reads only what it needs.

test_path_below_nullable_struct_field reads a leaf, a select, is_not_null of a leaf and a predicate through two nullable structs. It checks the results against in-memory evaluation and that no segment of the other fields is requested. Every case fails without the change.

Checked locally: cargo test -p vortex-layout --lib (629 passed), clippy with -D warnings, nightly fmt, and the byte counts above from a throwaway scan through a counting reader.

I used AI tools to help write and review this change.

@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Nov 13, 2026
… rest of the field

A nullable struct expands a field access `$.x` to `mask($.x, is_not_null($))`. For a
path below the field, such as `$.x.q`, the mask then sits between the path and the
field, so the partitioner hands the field's reader a bare `$` and the whole of `x`
is read. Mask the whole path instead.

On a file with `t: {x: {p, q}, r}`, both nullable, reading `$.t.x.q` or filtering on
`$.t.x.q > n` read 6.0 MB before and 160 KB after.

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
@gitRasheed
gitRasheed force-pushed the struct-path-below-nullable-field branch from 0b29aee to 5049e13 Compare November 19, 2026 20:57
@gitRasheed

gitRasheed commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

@robert3005 rebased onto develop. This is still the case there: expand_struct_root masks $.x before the partitioner sees $.x.q, so projecting or filtering on a field two nullable structs deep reads the whole inner struct. The nested pruning from #10957 runs into it for any struct inside a struct. Could you take a look when you get a chance please? ty!

connortsui20 added a commit that referenced this pull request Nov 19, 2026
On `develop`, a V2 scan that reads one field from a nullable struct can
still plan every child. The scan fails if a sibling uses a layout that
V2 cannot execute. This PR plans only the requested fields and the
validity needed to preserve their nulls.

For `address.zip`, the scan can skip `address.city` and an unused
sibling list. A null `address` still produces a null `zip`.

Related to #10952, which handles deep projections in the classic reader.

This change was implemented and reviewed with OpenAI Codex.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

/// Rebuilds a chain of `get_item` and `select` that starts at `$.field` on the field's expression
/// in `packed`, or returns `None` if `node` is not such a chain.
fn field_path(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this function seems wrong - we shouldn't be checking for particular expressions. We now have partitioner that takes layout and expression to do this.

@gitRasheed

Copy link
Copy Markdown
Contributor Author

Just saw #11638, which covers this for V2 scans. I guess this PR is moot, so I'll close it.

@gitRasheed gitRasheed closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants