Repository navigation
Push every Vortex-capable operation into the Spark file format - #11096
Draft
robert3005 wants to merge 9 commits into
Draft
robert3005 wants to merge 9 commits into
robert3005 wants to merge 9 commits into
Conversation
This was referenced Nov 3, 2026
robert3005
force-pushed
the
rk/sparkfileformat-pushdowns
branch
from
November 8, 2026 00:27
6ba3c70 to
5292913
Compare
This was referenced Nov 9, 2026
robert3005
force-pushed
the
rk/sparkfileformat-pushdowns
branch
3 times, most recently
from
November 12, 2026 10:21
083daa8 to
4a7cf7e
Compare
Reads now hand Spark's pushdowns to the native scan wherever Vortex can answer them: - Aggregates: COUNT(*), COUNT, MIN, MAX, and integer SUM (so AVG), grouped by any partition columns, answered from footer statistics or computed natively from only the columns involved (new NativeFiles.aggregate). Footer-only aggregates are answered on the Spark driver. - Runtime filters: dynamic pruning on partition columns drops files on the driver; on data columns readers evaluate the keys as a native set. An optimizer rule stops pruning subqueries from nesting through Vortex scans, which made TPC-DS q64 plan 122,883 subqueries. - LIMIT, OFFSET (whole files skipped through footer row counts), ORDER BY ... LIMIT (a shared, footer-seeded dynamic threshold through the new DynamicValue binding), and TABLESAMPLE as a row selection. - IN as a native set, LIKE patterns and column comparisons with no V1 form, nested struct pruning, and partition keys reported for v2 bucketing. - File splitting into row ranges, _metadata.row_index on the V1 path, footer column statistics for the cost-based optimizer, and sorted files reported to Spark. Other changes: - Spark 4.1 and 4.2 connector builds with their pushdowns. - Read and write Spark VARIANT columns as Vortex variants, and extract variant paths natively. - Record IsSorted only under editions that include it. - Send decimals of at most 18 digits to Spark as unscaled longs. - Scan metrics. SparkPushdownBenchmark measures each pushdown on and off. Signed-off-by: Robert Kruszewski <github@robertk.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Cleaner frees the native pointer of an Expression when the Expression becomes unreachable. The factories read the pointers of their inputs and then call native code. After the pointer reads the JIT can treat the inputs as unreachable, so a garbage collection during the call could free an input that native code still reads. A Spark TPC-DS run through Iceberg crashed with SIGSEGV in NativeExpression.literalList for a runtime-filter IN list. The factories, DynamicValue and DataSource.scan now keep their inputs reachable with Reference.reachabilityFence until the native call returns. Signed-off-by: Robert Kruszewski <github@robertk.io>
With 64 MiB splits, scans of tables of about 1 GB ran fewer tasks than there are cores. With 16 MiB on Spark 4.2, SF10 and 32 cores, TPC-H goes from 38.96 s to 36.3 s and TPC-DS from 98.4 s to 85.5 s. ClickBench does not change (52.72 s and 52.75 s). Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark sizes splits by an even share of the bytes per core. Vortex cuts files only at chunk boundaries, and packing cannot fill each split, so a file slightly larger than the share became two pieces and the scan took two waves of tasks. With 56 MB files of store_sales on 32 cores, a scan ran about 38 tasks. The share now gets a third more room. Signed-off-by: Robert Kruszewski <github@robertk.io>
Split planning read the layouts of all the columns of each file. For a scan of a few columns of a wide file, such as ClickBench hits with 105 columns, this used about 9% of the query time. VortexFile::splits_for and NativeFiles.splits now take the scanned columns. Signed-off-by: Robert Kruszewski <github@robertk.io>
Each task converts the runtime join filter of its scan. The filter can hold thousands of values, and one Java expression and native literal per value cost about 80 ms per task in TPC-DS q90. Integer and string lists now go to native code as one array. Signed-off-by: Robert Kruszewski <github@robertk.io>
On Java 17, Spark joins the partition files of a shuffle task with a memory map and unmap per file. The unmaps lock the address space of the executor, and concurrent Vortex scans wait on them. TPC-DS q31 on 32 cores took 1.15 s on Java 17 and 0.65 s on Java 21. Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark 4 enables ANSI rules by default, and the scan kept every SUM in Spark under them, because a wrapped 64-bit per-file sum cannot tell an overflow. A sum of TINYINT, SMALLINT, or INT values cannot overflow in a file of up to 2^32 rows, so these sums are now pushed under ANSI rules. A file with more rows fails the query with a request to disable aggregate pushdown. The driver merges footer partials with exact addition under ANSI rules. Spark states SUM(col + k) to a source only under ANSI rules. The scan answers it as SUM(col) + k * COUNT(col) when no value of the column plus the constant can overflow the type of the addition: a TINYINT or SMALLINT column plus an INT or BIGINT constant, or an INT column plus a BIGINT constant. ClickBench q30 sums SMALLINT ResolutionWidth plus 0 to 89. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
rk/sparkfileformat-pushdowns
branch
from
November 12, 2026 10:32
4a7cf7e to
9f922e9
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Push every operation that Vortex can answer into the Spark file format scan.
Changes
COUNT,MIN,MAX, integerSUM, grouped by partition columns, from footer statistics or native compute.LIMIT,OFFSET,ORDER BY ... LIMIT,TABLESAMPLE,IN,LIKE, nested struct pruning._metadata.row_index, footer statistics for the CBO, sorted file reporting.VARIANTread and write.SparkPushdownBenchmarkmeasures each pushdown on and off.API Changes
DynamicValue,FileAggregate,FileAggregateResult,NativeFiles.aggregate.Partition.scanArrow(BufferAllocator, boolean).