Skip to content

Make VortexFile::splits take a field mask and add it to the JNI bindings - #11490

Open
robert3005 wants to merge 2 commits into
developfrom
rk/jni-column-splits
Open

robert3005 wants to merge 2 commits into
developfrom
rk/jni-column-splits

Conversation

@robert3005

@robert3005 robert3005 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A scan of a few columns of a wide file does not need the splits of every column. To find them, the reader must read the layouts of all the columns, and for a wide table that costs more than the scan saves.

Changes

  • VortexFile::splits now takes a field mask and returns the natural row splits of those fields only. Callers pass FieldMask::All for the splits of every field. The Python bindings and the tests do that, so their behavior does not change.
  • NativeFiles.splits(session, readable) returns the split boundaries of the file behind a NativeReadable: split i holds rows [boundaries[i], boundaries[i + 1]).
  • NativeFiles.splits(session, readable, columns) does the same for a list of top-level columns. Columns that the file does not hold are ignored. If the file holds none of them, the splits follow every column.

VortexFile::splits_for returns the natural row splits of the fields in a
field mask. It reads only the layouts of those fields, so it is much
cheaper than splits() for a scan of a few columns of a wide file.

NativeFiles.splits exposes the splits of a file behind a NativeReadable,
for every column or for a list of top-level columns. An engine can then
plan splits that follow the chunks of the columns that it reads.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 added the changelog/feature A new feature label Nov 16, 2026
There is one method now, not splits and splits_for. A caller states
which fields it scans, and passes FieldMask::All for the splits of every
field.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 removed the changelog/feature A new feature label Nov 16, 2026
@robert3005 robert3005 changed the title Add column-aware row splits to VortexFile and the JNI bindings Make VortexFile::splits take a field mask and add it to the JNI bindings Nov 16, 2026
@robert3005 robert3005 added the changelog/break A breaking API change label Nov 16, 2026
@robert3005
robert3005 marked this pull request as ready for review November 16, 2026 11:36
@robert3005
robert3005 enabled auto-merge (squash) November 16, 2026 15:55

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant