Skip to content

fix(spark): reject unusable write options on the driver - #11151

Merged
robert3005 merged 2 commits into
vortex-data:developfrom
gitRasheed:spark-write-option-validation
Nov 20, 2026
Merged

robert3005 merged 2 commits into
vortex-data:developfrom
gitRasheed:spark-write-option-validation

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A malformed batch.size or vortex.write.batch.size, a negative vortex.workerThreads, or a vortex.session.provider class that can't be used fails only inside write tasks. On a V1 overwrite, Spark has already deleted the old files by then. The connector also ignores vortex.workerThreads once the JVM's worker pool is sized.

Changes

  • VortexOutputWriterFactory.validate checks the write options, and the factory's constructor calls it. Spark builds the factory on the driver before any task starts: in prepareWrite for the V1 format, and in toBatch for a V2 write. The existing vortex.write.compact check moves into it.
  • VortexSparkSession checks vortex.workerThreads on every call. It still sizes the pool once per JVM.
  • The driver loads the provider class without initializing or instantiating it, and rejects it only if no session could construct it: it doesn't implement VortexSessionProvider, is abstract, or has no no-argument constructor VortexSparkSession can call. A class the driver can't load is left to the executors, since a jar added with SparkContext.addJar reaches only them.

A batch size outside 1 to 65536 still falls back to the default with a warning, because rejecting it would turn a working V1 overwrite into one that deletes the old files and then fails.

Spark's V1 overwrite deletes the old files before any FileFormat hook sees the options, so this change can't keep them. The job now stops before launching tasks, with an error that names the option.

Tests

Each of these fails on develop:

  • VortexWriteOptionValidationTest.anUnusableOptionFailsAV1WriteOnTheDriver: a malformed batch size, a provider class that doesn't implement VortexSessionProvider and a negative worker count each make save() throw IllegalArgumentException. On develop the first two fail inside a task and surface as a SparkException, and the write ignores the third because an earlier write already sized the pool. The malformed vortex.write.compact case already passes on develop.
  • VortexWriteOptionValidationTest.anUnusableOptionFailsWhenAV2WriteIsPlanned: the same options make the V2 write's toBatch throw. DataFrameWriter sends file sources through the V1 format, so the test drives the table directly.
  • VortexSparkSessionTest.everyWorkerThreadCountIsCheckedAfterThePoolIsSized: after the pool is sized, VortexSparkSession.get still rejects a bad count, which covers reads as well as writes. Develop ignores it.

These call the new validate method, so they can't run on develop:

  • VortexSparkSessionTest.anUnusableProviderClassIsRejected: a class that doesn't implement VortexSessionProvider, an abstract provider and a provider with a private constructor are each rejected.
  • VortexSparkSessionTest.aProviderTheSessionCanConstructIsAccepted and aProviderTheDriverCannotLoadIsLeftToTheExecutors: a package-private provider and a class missing from the driver's classpath both pass validation, as they work on develop.
  • VortexSparkSessionTest.validatingAProviderNeitherInitializesNorConstructsIt: a provider counts runs of its static initializer and constructor. Validation runs neither, and resolving the session runs both.

VortexWriteOptionValidationTest.anOutOfRangeBatchSizeStillFallsBackToTheDefault passes on develop and guards the fallback.

Checked locally: :vortex-spark-3.5_2.12:test, :vortex-spark-3.5_2.13:test, :vortex-spark-4.0_2.13:test, spark41CompatTest, spark42CompatTest, spark43CompatTest and spotlessCheck.

I used AI tools to help write and review this change.

@gitRasheed
gitRasheed force-pushed the spark-write-option-validation branch from 884a270 to 852203c Compare November 3, 2026 15:19
A malformed batch.size or vortex.write.batch.size, a negative
vortex.workerThreads, or a session provider class that cannot be used
failed only inside write tasks, after a V1 overwrite had already deleted
the old files. vortex.workerThreads also went unchecked once the JVM's
worker pool was sized.

The output writer factory now checks these options, together with
vortex.write.compact, when Spark builds it on the driver, which is
before any task starts, for V1 writes and for V2 writes in toBatch.
Every vortex.workerThreads
value is checked even though the pool is still sized once. The driver
loads the provider class without initializing or instantiating it, and
rejects it only if no session could construct it. A class the driver
cannot load is left to the executors, since a jar added with addJar
reaches only them.

An out-of-range batch size still falls back to the default with a
warning. Rejecting it would turn a working V1 overwrite into one that
deletes the old files and then fails.

Spark's V1 overwrite deletes the old files before any FileFormat hook
sees the options, so on that path the files are still lost. The job now
stops before launching tasks and names the option.

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
@gitRasheed
gitRasheed force-pushed the spark-write-option-validation branch from 852203c to 8196d70 Compare November 19, 2026 22:41
@gitRasheed

Copy link
Copy Markdown
Contributor Author

Rebased onto develop. The vortex.write.compact check from #11406 now goes through the same validate. I also narrowed the provider check so it can't reject a provider that works today: the driver only rejects a class no session could construct, and leaves one it can't load to the executors, since a jar added with addJar only reaches them.

@robert3005 robert3005 added the changelog/fix A bug fix label Nov 20, 2026
configureRuntime set its flag before it sized the pool, so a task that
started at the same time could skip the sizing and run on a pool that
did not have its configured size yet. The flag is now set under a lock
after setWorkerThreads returns, and a failed sizing lets the next caller
try again.

Also make VortexOutputWriterFactory.validate private, add a test for a
malformed vortex.write.batch.size and parameterize the worker-thread
test.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005

Copy link
Copy Markdown
Contributor

Thanks, I have touched it up a little bit to fix a bug that was there before

@robert3005
robert3005 merged commit e750a65 into vortex-data:develop Nov 20, 2026
74 checks passed
ChristopherWirt added a commit to quantcast/vortex that referenced this pull request Nov 20, 2026
Merge vortex-data/vortex develop at 79e75c1.

Conflict resolutions:
- zone_map.rs: keep the fork's large IN-list pruning tests; upstream moved
  the other zone map tests to vortex-scan.
- plan/tests.rs: take upstream; port the fork's fallible ListLayout::try_new
  to the moved lowering tests.
- VortexSparkSession: take upstream's vortex-data#11151, which supersedes the fork's
  version of the same change.
- VortexOutputWriterFactory: keep the fork's public validate covering
  batch.bytes, capacity reset and listLayout.
- VortexWriteOptionValidationTest: upstream V1/V2 structure with the fork's
  option cases; out-of-range batch sizes stay rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants