Repository navigation
feat(jni): let Java and Spark writers use the compact strategy - #11406
Merged
robert3005 merged 1 commit intoNov 12, 2026
Merged
Conversation
The JNI writer always used the default write strategy, so Java and Spark users could not pick the compact one that Python selects with VortexWriteOptions.compact() and that vx convert exposes as --strategy compact. Add VortexWriter.Builder#compact(boolean) and a vortex.write.compact Spark option, both off by default. The Spark option is parsed when the factory is built on the driver, so a malformed value fails once there rather than in every task. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
robert3005
approved these changes
Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The JNI writer always uses the default write strategy, so Java and Spark users cannot select the compact one that Python (
VortexWriteOptions.compact()) andvx convert --strategy compactalready offer. #11374 measures 169.95 MB default vs 106.52 MB compact (37% smaller) on 3M rows.Closes #11374. Covers the compression part of #7725.
Changes
vortex-jni:createandcreateStreamtake acompactflag, which builds the strategy the way Python does.VortexWriter.Builder#compact(boolean), off by default.vortex.write.compactoption, parsed on the driver inVortexOutputWriterFactory, so a malformed value fails once there rather than in every task. Documented inspark.md.Tested with:
JNIWriterTest#testCompactWritesASmallerFile(URI and stream sinks) andVortexDataSourceWriteTest#testCompactWriteIsSmallerAndReadsBack: compact output is smaller and reads back equalVortexOutputWriterTest: malformed option fails at factory buildcargo clippy -p vortex-jni -- -D warnings,spotlessCheck,javadocAI assistance
Written with agentic AI assistance.