Repository navigation
perf(spark): append rows straight into the Arrow batch - #11152
Merged
robert3005 merged 1 commit intoNov 3, 2026
Merged
Conversation
gitRasheed
force-pushed
the
spark-writer-direct-arrow
branch
from
November 3, 2026 15:32
da1840f to
77bb1c1
Compare
VortexOutputWriter copied every row into a list and converted the list to Arrow value by value when the batch was full. Each row is now appended straight into the Arrow vectors through one typed field writer per column, built once from the schema. Strings are copied from the row into the Arrow buffer without a byte[] in between, and decimals of up to 18 digits are written from the unscaled long. A null slot is left as allocated, so nothing is written below a null struct, as before; a struct catches its children's counts up to its own slot when it next gets a value. This relies on fresh vectors for every batch: Arrow zeroes their validity and offsets, and fills the offsets of skipped slots at the next value or at setValueCount. Vectors start at 64 slots and grow, so a small batch no longer takes Arrow's default capacity while the native writer holds it. When Arrow memory is still allocated at close, the writer now logs a warning and leaves the allocator open, so a buffer the native side releases later is still freed. It used to close the allocator and log at debug, and a release into the closed allocator then failed. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
gitRasheed
force-pushed
the
spark-writer-direct-arrow
branch
from
November 3, 2026 15:59
77bb1c1 to
17f7c55
Compare
gitRasheed
marked this pull request as ready for review
November 3, 2026 15:59
robert3005
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VortexOutputWritercopied every row (row.copy()) into a list, and once a batch was full it converted the list to Arrow one value at a time, dispatching on the Spark type for every value. Now each row goes straight into the batch's Arrow vectors as it arrives. One typedArrowFieldWriterper column, built from the schema once, does the appending.Changes
ArrowFieldWriter, which works like Spark's ownArrowWriter. That class can't be used here because the connector relocates Arrow. Every writer counts what it has appended, so list elements and map entries find their next slot without asking the vector. Strings are copied from the row into the Arrow buffer with nobyte[]in between. Decimals of up to 18 digits are written from the unscaled long.setValueCount.write()appends the row to the vectors and hands the batch over once it is full. Because Spark reuses the row object, the row is fully consumed beforewrite()returns. If a write fails part-way through a row, the batch's columns are misaligned, so that failure counts as a failed write. Upstream's abort handling stays as it is: later writes are refused, andclose()drops the last batch and logs instead of throwing.close(), the writer logs a warning and leaves the allocator open, so a buffer the native side releases later still gets freed. Before, it closed the allocator anyway and logged at debug. A later release into the closed allocator then failed and the memory was never freed.VortexOutputWriterAbortTest, the oversized row now fails inwrite()at every batch size and never reaches theclose()that flushes the batch. The case that put an oversized row in every partition is removed: the first one fails the task, so the rest never reach their writers.I timed it with a release JNI build, writing 100,000 rows straight through one
VortexOutputWriter(no Spark job) at the default batch size. The schema is wide and nested: 8 top-level leaves, a 24-leaf struct, a 16-leaf struct with a nested 16-leaf struct, an array of 8-leaf structs, a map and an array of strings. I took the median of 5 runs after 2 warm-ups, and ran before and after twice, alternating. The file bytes are the same before and after.Process CPU includes the native workers and is the noisiest of the three. The timing harness isn't committed.
Robert's draft #11096 also edits this writer: it adds variant handling and a
setNullhelper topopulateVector, and this PR removes that method. With this change, that logic would go in a variantArrowFieldWriter.Tests
VortexDataSourceWriteTest.testNullsAtEveryNestingLevelRoundTrip: every supported type (the test checks thatfreally is aFloatType), plus structs, arrays of structs, nested arrays and maps of structs, with nulls at every level, round-trips through Spark at batch sizes 1, 300 and 65536. One struct column is null in runs of 300, so at batch size 300 some batches are all null, some have no nulls, and some switch part-way. Fails without the change: at batch size 1, every one-row batch takes Arrow's default capacity, and the test JVM runs out of its 512 MiB of direct memory.VortexOutputWriterTest.consumesReusedRowsAtWriteTime: rows are rewritten or freed as soon aswrite()returns, including off-heap rows and a reusedbyte[]. It guards droppingrow.copy(). It also fails without the change:copy()of aGenericInternalRowkeeps the caller'sbyte[], so some binary values come back overwritten. Spark itself passesUnsafeRows, which copy deeply.memoryGrowsWithRowsNotBatches: 400 one-row batches hold less than 4 MiB. Fails without the change: each batch takes Arrow's default capacity.nullStructsAllocateNothingBelowThem: 4,000 rows whose 32-leaf struct is null hold less than 256 KiB, because the leaves are never written. Fails if a null struct writes a null into each leaf, which grows every leaf to the batch's size. It passes on develop, which allocates no Arrow memory until a batch is full.aRowThatFailsPartWayDiscardsTheBatch: the first column of a row is appended, then the second fails. Later writes are refused, andclose()drops the pending batch and frees all its Arrow memory. Fails without the change, where the bad value is only read when the batch is converted, so the failingwrite()doesn't throw.aLateReleaseDoesNotFailCloseAndIsStillFreed: a buffer released afterclose()is freed, andclose()succeeds. Fails without the change.Checked locally:
:vortex-spark-3.5_2.12:test,:vortex-spark-3.5_2.13:test,:vortex-spark-4.0_2.13:test,spark41CompatTest,spark42CompatTestandspotlessCheckpass.I used AI tools to help write and review this change.