Repository navigation
fix(spark): reject unusable write options on the driver - #11151
Merged
robert3005 merged 2 commits intoNov 20, 2026
Merged
Conversation
gitRasheed
force-pushed
the
spark-write-option-validation
branch
from
November 3, 2026 15:19
884a270 to
852203c
Compare
A malformed batch.size or vortex.write.batch.size, a negative vortex.workerThreads, or a session provider class that cannot be used failed only inside write tasks, after a V1 overwrite had already deleted the old files. vortex.workerThreads also went unchecked once the JVM's worker pool was sized. The output writer factory now checks these options, together with vortex.write.compact, when Spark builds it on the driver, which is before any task starts, for V1 writes and for V2 writes in toBatch. Every vortex.workerThreads value is checked even though the pool is still sized once. The driver loads the provider class without initializing or instantiating it, and rejects it only if no session could construct it. A class the driver cannot load is left to the executors, since a jar added with addJar reaches only them. An out-of-range batch size still falls back to the default with a warning. Rejecting it would turn a working V1 overwrite into one that deletes the old files and then fails. Spark's V1 overwrite deletes the old files before any FileFormat hook sees the options, so on that path the files are still lost. The job now stops before launching tasks and names the option. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
gitRasheed
force-pushed
the
spark-write-option-validation
branch
from
November 19, 2026 22:41
852203c to
8196d70
Compare
Contributor
Author
|
Rebased onto develop. The |
configureRuntime set its flag before it sized the pool, so a task that started at the same time could skip the sizing and run on a pool that did not have its configured size yet. The flag is now set under a lock after setWorkerThreads returns, and a failed sizing lets the next caller try again. Also make VortexOutputWriterFactory.validate private, add a test for a malformed vortex.write.batch.size and parameterize the worker-thread test. Signed-off-by: Robert Kruszewski <github@robertk.io>
Contributor
|
Thanks, I have touched it up a little bit to fix a bug that was there before |
robert3005
approved these changes
Nov 20, 2026
robert3005
approved these changes
Nov 20, 2026
ChristopherWirt
added a commit
to quantcast/vortex
that referenced
this pull request
Nov 20, 2026
Merge vortex-data/vortex develop at 79e75c1. Conflict resolutions: - zone_map.rs: keep the fork's large IN-list pruning tests; upstream moved the other zone map tests to vortex-scan. - plan/tests.rs: take upstream; port the fork's fallible ListLayout::try_new to the moved lowering tests. - VortexSparkSession: take upstream's vortex-data#11151, which supersedes the fork's version of the same change. - VortexOutputWriterFactory: keep the fork's public validate covering batch.bytes, capacity reset and listLayout. - VortexWriteOptionValidationTest: upstream V1/V2 structure with the fork's option cases; out-of-range batch sizes stay rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A malformed
batch.sizeorvortex.write.batch.size, a negativevortex.workerThreads, or avortex.session.providerclass that can't be used fails only inside write tasks. On a V1 overwrite, Spark has already deleted the old files by then. The connector also ignoresvortex.workerThreadsonce the JVM's worker pool is sized.Changes
VortexOutputWriterFactory.validatechecks the write options, and the factory's constructor calls it. Spark builds the factory on the driver before any task starts: inprepareWritefor the V1 format, and intoBatchfor a V2 write. The existingvortex.write.compactcheck moves into it.VortexSparkSessionchecksvortex.workerThreadson every call. It still sizes the pool once per JVM.VortexSessionProvider, is abstract, or has no no-argument constructorVortexSparkSessioncan call. A class the driver can't load is left to the executors, since a jar added withSparkContext.addJarreaches only them.A batch size outside 1 to 65536 still falls back to the default with a warning, because rejecting it would turn a working V1 overwrite into one that deletes the old files and then fails.
Spark's V1 overwrite deletes the old files before any
FileFormathook sees the options, so this change can't keep them. The job now stops before launching tasks, with an error that names the option.Tests
Each of these fails on develop:
VortexWriteOptionValidationTest.anUnusableOptionFailsAV1WriteOnTheDriver: a malformed batch size, a provider class that doesn't implementVortexSessionProviderand a negative worker count each makesave()throwIllegalArgumentException. On develop the first two fail inside a task and surface as aSparkException, and the write ignores the third because an earlier write already sized the pool. The malformedvortex.write.compactcase already passes on develop.VortexWriteOptionValidationTest.anUnusableOptionFailsWhenAV2WriteIsPlanned: the same options make the V2 write'stoBatchthrow.DataFrameWritersends file sources through the V1 format, so the test drives the table directly.VortexSparkSessionTest.everyWorkerThreadCountIsCheckedAfterThePoolIsSized: after the pool is sized,VortexSparkSession.getstill rejects a bad count, which covers reads as well as writes. Develop ignores it.These call the new
validatemethod, so they can't run on develop:VortexSparkSessionTest.anUnusableProviderClassIsRejected: a class that doesn't implementVortexSessionProvider, an abstract provider and a provider with a private constructor are each rejected.VortexSparkSessionTest.aProviderTheSessionCanConstructIsAcceptedandaProviderTheDriverCannotLoadIsLeftToTheExecutors: a package-private provider and a class missing from the driver's classpath both pass validation, as they work on develop.VortexSparkSessionTest.validatingAProviderNeitherInitializesNorConstructsIt: a provider counts runs of its static initializer and constructor. Validation runs neither, and resolving the session runs both.VortexWriteOptionValidationTest.anOutOfRangeBatchSizeStillFallsBackToTheDefaultpasses on develop and guards the fallback.Checked locally:
:vortex-spark-3.5_2.12:test,:vortex-spark-3.5_2.13:test,:vortex-spark-4.0_2.13:test,spark41CompatTest,spark42CompatTest,spark43CompatTestandspotlessCheck.I used AI tools to help write and review this change.