Repository navigation
perf(layout): read a path below a nullable struct's field without the rest of the field - #10952
gitRasheed wants to merge 1 commit into
Conversation
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
… rest of the field
A nullable struct expands a field access `$.x` to `mask($.x, is_not_null($))`. For a
path below the field, such as `$.x.q`, the mask then sits between the path and the
field, so the partitioner hands the field's reader a bare `$` and the whole of `x`
is read. Mask the whole path instead.
On a file with `t: {x: {p, q}, r}`, both nullable, reading `$.t.x.q` or filtering on
`$.t.x.q > n` read 6.0 MB before and 160 KB after.
Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
0b29aee to
5049e13
Compare
|
@robert3005 rebased onto develop. This is still the case there: |
On `develop`, a V2 scan that reads one field from a nullable struct can still plan every child. The scan fails if a sibling uses a layout that V2 cannot execute. This PR plans only the requested fields and the validity needed to preserve their nulls. For `address.zip`, the scan can skip `address.city` and an unused sibling list. A null `address` still produces a null `zip`. Related to #10952, which handles deep projections in the classic reader. This change was implemented and reviewed with OpenAI Codex. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
|
|
||
| /// Rebuilds a chain of `get_item` and `select` that starts at `$.field` on the field's expression | ||
| /// in `packed`, or returns `None` if `node` is not such a chain. | ||
| fn field_path( |
There was a problem hiding this comment.
this function seems wrong - we shouldn't be checking for particular expressions. We now have partitioner that takes layout and expression to do this.
|
Just saw #11638, which covers this for V2 scans. I guess this PR is moot, so I'll close it. |
Summary
Reading a field two levels into nullable structs read the whole inner struct. For
t: {x: {p, q}, r}withtandxnullable, projecting$.t.x.qor filtering on$.t.x.q > nread 6.05 MB of a 23.7 MB file, as much as reading all ofx. With this change it reads 160 KB. One level,$.s.b, was already pruned.expand_struct_rootrewrites a field access$.xon a nullable struct tomask($.x, is_not_null($)), so the field keeps the struct's nulls. For$.x.qthat givesget_item(q, mask($.x, ...)), and the mask hides the path from the partitioner:x's reader gets$and reads every field. A Spark connector that pushes nested schema pruning down runs into this for any struct inside a nullable struct, since Spark structs are nearly always nullable.Changes
A
get_item/selectchain that starts at$.fieldis rewritten as a whole tomask(chain_on_field, is_not_null($)), so the field's reader gets the path and reads only what it needs.test_path_below_nullable_struct_fieldreads a leaf, aselect,is_not_nullof a leaf and a predicate through two nullable structs. It checks the results against in-memory evaluation and that no segment of the other fields is requested. Every case fails without the change.Checked locally:
cargo test -p vortex-layout --lib(629 passed), clippy with-D warnings, nightly fmt, and the byte counts above from a throwaway scan through a counting reader.I used AI tools to help write and review this change.