Repository navigation
Faster Spark reads and smaller JNI and stats fixes - #11109
Merged
Merged
Conversation
Merging this PR will not alter performance1 warning
Comparing |
robert3005
force-pushed
the
rk/spark-jni-improvements
branch
from
November 3, 2026 11:41
8ab69fe to
37d7ff7
Compare
robert3005
enabled auto-merge (squash)
November 3, 2026 11:41
These changes stand on their own and need no new Vortex or JNI API: - Read Spark columns straight from the Arrow buffers: validity bits, fixed-width values, string offsets and string views, and decimals of at most 18 digits as their unscaled long. Turn off the per-value bounds and null checks of the relocated Arrow, as Spark does for its own Arrow, unless the user sets them. - Broadcast the Hadoop configuration to readers once instead of shipping it with every task. - Push IN down as a native set rather than an OR of equalities. A null element stays in the set, so NOT IN keeps SQL's three-valued result. - The JNI writer takes the exported Arrow schemas it receives and releases them, so a batch written through writeBatch no longer leaks the schema. - Skip file statistics and zone maps for variants nested in structs, lists, and maps, which no builder can hold. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
rk/spark-jni-improvements
branch
from
November 3, 2026 20:51
37d7ff7 to
9411de2
Compare
myrrc
approved these changes
Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ArrowPropertiesturns off the bounds and null checks of the relocated Arrow on each value, as Spark does for its own Arrow, unless the user sets them.VortexIo.broadcast), instead of with every task.INis pushed down as a native set instead of anORof equalities. A null element stays in the set, soNOT INkeeps the SQL result.writeBatchno longer leaks the exported schema. The Java release added in fix(jni): release the schema export in VortexWriter.build() #11081 now runs only when native code fails before it takes the schema.