Skip to content

feat(jni): let Java and Spark writers use the compact strategy - #11406

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:feat/jni-compact-write
Nov 12, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:feat/jni-compact-write

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Summary

The JNI writer always uses the default write strategy, so Java and Spark users cannot select the compact one that Python (VortexWriteOptions.compact()) and vx convert --strategy compact already offer. #11374 measures 169.95 MB default vs 106.52 MB compact (37% smaller) on 3M rows.

Closes #11374. Covers the compression part of #7725.

Changes

  • vortex-jni: create and createStream take a compact flag, which builds the strategy the way Python does.
  • VortexWriter.Builder#compact(boolean), off by default.
  • Spark: vortex.write.compact option, parsed on the driver in VortexOutputWriterFactory, so a malformed value fails once there rather than in every task. Documented in spark.md.

Tested with:

  • JNIWriterTest#testCompactWritesASmallerFile (URI and stream sinks) and VortexDataSourceWriteTest#testCompactWriteIsSmallerAndReadsBack: compact output is smaller and reads back equal
  • VortexOutputWriterTest: malformed option fails at factory build
  • cargo clippy -p vortex-jni -- -D warnings, spotlessCheck, javadoc

AI assistance

Written with agentic AI assistance.

The JNI writer always used the default write strategy, so Java and Spark users could not pick the compact one that Python selects with VortexWriteOptions.compact() and that vx convert exposes as --strategy compact. Add VortexWriter.Builder#compact(boolean) and a vortex.write.compact Spark option, both off by default. The Spark option is parsed when the factory is built on the driver, so a malformed value fails once there rather than in every task.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@robert3005
robert3005 enabled auto-merge (squash) November 12, 2026 10:23
@robert3005
robert3005 merged commit dcada48 into vortex-data:develop Nov 12, 2026
97 of 99 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vortex-jni: expose compact write options (VortexWriteOptions.compact())

2 participants