Skip to content

perf(spark): append rows straight into the Arrow batch - #11152

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
gitRasheed:spark-writer-direct-arrow
Nov 3, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
gitRasheed:spark-writer-direct-arrow

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

VortexOutputWriter copied every row (row.copy()) into a list, and once a batch was full it converted the list to Arrow one value at a time, dispatching on the Spark type for every value. Now each row goes straight into the batch's Arrow vectors as it arrives. One typed ArrowFieldWriter per column, built from the schema once, does the appending.

Changes

  • New ArrowFieldWriter, which works like Spark's own ArrowWriter. That class can't be used here because the connector relocates Arrow. Every writer counts what it has appended, so list elements and map entries find their next slot without asking the vector. Strings are copied from the row into the Arrow buffer with no byte[] in between. Decimals of up to 18 digits are written from the unscaled long.
  • A null slot is left as allocated, so nothing is written below a null struct, the same as before. A struct catches its children's counts up to its own slot when it next gets a value. This relies on fresh vectors for every batch: Arrow zeroes their validity and offsets, and fills the offsets of skipped slots at the next value or at setValueCount.
  • write() appends the row to the vectors and hands the batch over once it is full. Because Spark reuses the row object, the row is fully consumed before write() returns. If a write fails part-way through a row, the batch's columns are misaligned, so that failure counts as a failed write. Upstream's abort handling stays as it is: later writes are refused, and close() drops the last batch and logs instead of throwing.
  • Vectors start at 64 slots and grow as needed. A batch of a few rows no longer takes Arrow's default capacity of several thousand values while the native writer holds it.
  • If Arrow memory is still allocated at close(), the writer logs a warning and leaves the allocator open, so a buffer the native side releases later still gets freed. Before, it closed the allocator anyway and logged at debug. A later release into the closed allocator then failed and the memory was never freed.
  • In VortexOutputWriterAbortTest, the oversized row now fails in write() at every batch size and never reaches the close() that flushes the batch. The case that put an oversized row in every partition is removed: the first one fails the task, so the rest never reach their writers.

I timed it with a release JNI build, writing 100,000 rows straight through one VortexOutputWriter (no Spark job) at the default batch size. The schema is wide and nested: 8 top-level leaves, a 24-leaf struct, a 16-leaf struct with a nested 16-leaf struct, an array of 8-leaf structs, a map and an array of strings. I took the median of 5 runs after 2 warm-ups, and ran before and after twice, alternating. The file bytes are the same before and after.

Struct columns null Task-thread CPU before After Wall before After Process CPU before After
10% 694 ms 387 ms (-44%) 772 ms 455 ms (-41%) 1,630 ms 1,245 ms (-24%)
90% 261 ms 204 ms (-22%) 305 ms 249 ms (-18%) 855 ms 770 ms (-10%)

Process CPU includes the native workers and is the noisiest of the three. The timing harness isn't committed.

Robert's draft #11096 also edits this writer: it adds variant handling and a setNull helper to populateVector, and this PR removes that method. With this change, that logic would go in a variant ArrowFieldWriter.

Tests

  • VortexDataSourceWriteTest.testNullsAtEveryNestingLevelRoundTrip: every supported type (the test checks that f really is a FloatType), plus structs, arrays of structs, nested arrays and maps of structs, with nulls at every level, round-trips through Spark at batch sizes 1, 300 and 65536. One struct column is null in runs of 300, so at batch size 300 some batches are all null, some have no nulls, and some switch part-way. Fails without the change: at batch size 1, every one-row batch takes Arrow's default capacity, and the test JVM runs out of its 512 MiB of direct memory.
  • VortexOutputWriterTest.consumesReusedRowsAtWriteTime: rows are rewritten or freed as soon as write() returns, including off-heap rows and a reused byte[]. It guards dropping row.copy(). It also fails without the change: copy() of a GenericInternalRow keeps the caller's byte[], so some binary values come back overwritten. Spark itself passes UnsafeRows, which copy deeply.
  • memoryGrowsWithRowsNotBatches: 400 one-row batches hold less than 4 MiB. Fails without the change: each batch takes Arrow's default capacity.
  • nullStructsAllocateNothingBelowThem: 4,000 rows whose 32-leaf struct is null hold less than 256 KiB, because the leaves are never written. Fails if a null struct writes a null into each leaf, which grows every leaf to the batch's size. It passes on develop, which allocates no Arrow memory until a batch is full.
  • aRowThatFailsPartWayDiscardsTheBatch: the first column of a row is appended, then the second fails. Later writes are refused, and close() drops the pending batch and frees all its Arrow memory. Fails without the change, where the bad value is only read when the batch is converted, so the failing write() doesn't throw.
  • aLateReleaseDoesNotFailCloseAndIsStillFreed: a buffer released after close() is freed, and close() succeeds. Fails without the change.

Checked locally: :vortex-spark-3.5_2.12:test, :vortex-spark-3.5_2.13:test, :vortex-spark-4.0_2.13:test, spark41CompatTest, spark42CompatTest and spotlessCheck pass.

I used AI tools to help write and review this change.

@gitRasheed
gitRasheed force-pushed the spark-writer-direct-arrow branch from da1840f to 77bb1c1 Compare November 3, 2026 15:32
VortexOutputWriter copied every row into a list and converted the list
to Arrow value by value when the batch was full. Each row is now
appended straight into the Arrow vectors through one typed field writer
per column, built once from the schema. Strings are copied from the row
into the Arrow buffer without a byte[] in between, and decimals of up to
18 digits are written from the unscaled long.

A null slot is left as allocated, so nothing is written below a null
struct, as before; a struct catches its children's counts up to its own
slot when it next gets a value. This relies on fresh vectors for every
batch: Arrow zeroes their validity and offsets, and fills the offsets of
skipped slots at the next value or at setValueCount.

Vectors start at 64 slots and grow, so a small batch no longer takes
Arrow's default capacity while the native writer holds it.

When Arrow memory is still allocated at close, the writer now logs a
warning and leaves the allocator open, so a buffer the native side
releases later is still freed. It used to close the allocator and log
at debug, and a release into the closed allocator then failed.

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
@gitRasheed
gitRasheed force-pushed the spark-writer-direct-arrow branch from 77bb1c1 to 17f7c55 Compare November 3, 2026 15:59
@gitRasheed
gitRasheed marked this pull request as ready for review November 3, 2026 15:59
@robert3005 robert3005 added the changelog/performance A performance improvement label Nov 3, 2026
@robert3005
robert3005 merged commit 1012d0c into vortex-data:develop Nov 3, 2026
153 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants