Skip to content

Faster Spark reads and smaller JNI and stats fixes - #11109

Merged
connortsui20 merged 1 commit into
developfrom
rk/spark-jni-improvements
Nov 4, 2026
Merged

connortsui20 merged 1 commit into
developfrom
rk/spark-jni-improvements

Conversation

@robert3005

@robert3005 robert3005 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor
  • Spark columns read their values directly from the Arrow buffers: validity bits, fixed-width values, string offsets and string views. A decimal of 18 digits or fewer is returned as its unscaled long, as Spark's own vectorized readers do.
  • ArrowProperties turns off the bounds and null checks of the relocated Arrow on each value, as Spark does for its own Arrow, unless the user sets them.
  • Readers get the Hadoop configuration through a broadcast (VortexIo.broadcast), instead of with every task.
  • IN is pushed down as a native set instead of an OR of equalities. A null element stays in the set, so NOT IN keeps the SQL result.
  • The JNI writer takes ownership of the Arrow schemas it receives and releases them, so writeBatch no longer leaks the exported schema. The Java release added in fix(jni): release the schema export in VortexWriter.build() #11081 now runs only when native code fails before it takes the schema.
  • File statistics and zone maps skip variants nested in structs, lists and maps, which no builder can hold.

@robert3005 robert3005 added the changelog/performance A performance improvement label Nov 2, 2026
@codspeed

codspeed Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 1878 untouched benchmarks


Comparing rk/spark-jni-improvements (9411de2) with develop (1012d0c)

Open in CodSpeed

These changes stand on their own and need no new Vortex or JNI API:

- Read Spark columns straight from the Arrow buffers: validity bits,
  fixed-width values, string offsets and string views, and decimals of at
  most 18 digits as their unscaled long. Turn off the per-value bounds and
  null checks of the relocated Arrow, as Spark does for its own Arrow,
  unless the user sets them.
- Broadcast the Hadoop configuration to readers once instead of shipping
  it with every task.
- Push IN down as a native set rather than an OR of equalities. A null
  element stays in the set, so NOT IN keeps SQL's three-valued result.
- The JNI writer takes the exported Arrow schemas it receives and releases
  them, so a batch written through writeBatch no longer leaks the schema.
- Skip file statistics and zone maps for variants nested in structs,
  lists, and maps, which no builder can hold.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 force-pushed the rk/spark-jni-improvements branch from 37d7ff7 to 9411de2 Compare November 3, 2026 20:51
@robert3005
robert3005 requested a review from myrrc November 4, 2026 11:39
@connortsui20
connortsui20 disabled auto-merge November 4, 2026 14:56
@connortsui20
connortsui20 merged commit fdd96d0 into develop Nov 4, 2026
73 checks passed
@connortsui20
connortsui20 deleted the rk/spark-jni-improvements branch November 4, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants