Repository navigation
Spark: let Spark merge subqueries that scan one Vortex table - #11330
Draft
robert3005 wants to merge 11 commits into
Draft
robert3005 wants to merge 11 commits into
robert3005 wants to merge 11 commits into
Conversation
Reads now hand Spark's pushdowns to the native scan wherever Vortex can answer them: - Aggregates: COUNT(*), COUNT, MIN, MAX, and integer SUM (so AVG), grouped by any partition columns, answered from footer statistics or computed natively from only the columns involved (new NativeFiles.aggregate). Footer-only aggregates are answered on the Spark driver. - Runtime filters: dynamic pruning on partition columns drops files on the driver; on data columns readers evaluate the keys as a native set. An optimizer rule stops pruning subqueries from nesting through Vortex scans, which made TPC-DS q64 plan 122,883 subqueries. - LIMIT, OFFSET (whole files skipped through footer row counts), ORDER BY ... LIMIT (a shared, footer-seeded dynamic threshold through the new DynamicValue binding), and TABLESAMPLE as a row selection. - IN as a native set, LIKE patterns and column comparisons with no V1 form, nested struct pruning, and partition keys reported for v2 bucketing. - File splitting into row ranges, _metadata.row_index on the V1 path, footer column statistics for the cost-based optimizer, and sorted files reported to Spark. Other changes: - Spark 4.1 and 4.2 connector builds with their pushdowns. - Read and write Spark VARIANT columns as Vortex variants, and extract variant paths natively. - Record IsSorted only under editions that include it. - Send decimals of at most 18 digits to Spark as unscaled longs. - Scan metrics. SparkPushdownBenchmark measures each pushdown on and off. Signed-off-by: Robert Kruszewski <github@robertk.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ines A column that the dictionary layout declines, such as a decimal with many distinct values, can still have parts with few distinct values, such as its integer bytes. The fallback path now uses the compressor with IntDictScheme. For TPC-H SF10 l_quantity this reduces the column from 97.6 MB to 45.2 MB. Signed-off-by: Robert Kruszewski <github@robertk.io>
A Cleaner frees the native pointer of an Expression when the Expression becomes unreachable. The factories read the pointers of their inputs and then call native code. After the pointer reads the JIT can treat the inputs as unreachable, so a garbage collection during the call could free an input that native code still reads. A Spark TPC-DS run through Iceberg crashed with SIGSEGV in NativeExpression.literalList for a runtime-filter IN list. The factories, DynamicValue and DataSource.scan now keep their inputs reachable with Reference.reachabilityFence until the native call returns. Signed-off-by: Robert Kruszewski <github@robertk.io>
With 64 MiB splits, scans of tables of about 1 GB ran fewer tasks than there are cores. With 16 MiB on Spark 4.2, SF10 and 32 cores, TPC-H goes from 38.96 s to 36.3 s and TPC-DS from 98.4 s to 85.5 s. ClickBench does not change (52.72 s and 52.75 s). Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark sizes splits by an even share of the bytes per core. Vortex cuts files only at chunk boundaries, and packing cannot fill each split, so a file slightly larger than the share became two pieces and the scan took two waves of tasks. With 56 MB files of store_sales on 32 cores, a scan ran about 38 tasks. The share now gets a third more room. Signed-off-by: Robert Kruszewski <github@robertk.io>
Split planning read the layouts of all the columns of each file. For a scan of a few columns of a wide file, such as ClickBench hits with 105 columns, this used about 9% of the query time. VortexFile::splits_for and NativeFiles.splits now take the scanned columns. Signed-off-by: Robert Kruszewski <github@robertk.io>
Each task converts the runtime join filter of its scan. The filter can hold thousands of values, and one Java expression and native literal per value cost about 80 ms per task in TPC-DS q90. Integer and string lists now go to native code as one array. Signed-off-by: Robert Kruszewski <github@robertk.io>
On Java 17, Spark joins the partition files of a shuffle task with a memory map and unmap per file. The unmaps lock the address space of the executor, and concurrent Vortex scans wait on them. TPC-DS q31 on 32 cores took 1.15 s on Java 17 and 0.65 s on Java 21. Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark 4 enables ANSI rules by default, and the scan kept every SUM in Spark under them, because a wrapped 64-bit per-file sum cannot tell an overflow. A sum of TINYINT, SMALLINT, or INT values cannot overflow in a file of up to 2^32 rows, so these sums are now pushed under ANSI rules. A file with more rows fails the query with a request to disable aggregate pushdown. The driver merges footer partials with exact addition under ANSI rules. Spark states SUM(col + k) to a source only under ANSI rules. The scan answers it as SUM(col) + k * COUNT(col) when no value of the column plus the constant can overflow the type of the addition: a TINYINT or SMALLINT column plus an INT or BIGINT constant, or an INT column plus a BIGINT constant. ClickBench q30 sums SMALLINT ResolutionWidth plus 0 to 89. Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark merges scalar subqueries with the same filters into one scan only when their scans are equal. A DataSource V2 scan holds only the columns of its subquery, so the scans of TPC-DS q9 differ and Spark reads store_sales 15 times. Spark's own Parquet V2 reader has the same problem. The session extensions now add two rules. Before Spark merges subqueries, Widen changes the Vortex scans that differ only in their columns to read the union of their columns, below a projection to their own columns. After the merge, Narrow changes each scan that Spark did not merge back to its own columns. On TPC-DS SF10 q9 the Vortex scans go from 15 to 5, as on the V1 path. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
force-pushed
the
rk/sparkfileformat-pushdowns
branch
4 times, most recently
from
November 12, 2026 10:32
4a7cf7e to
9f922e9
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #11096.
Problem
Spark merges scalar subqueries that have the same filters into one scan. It does this only when their scans are equal. Spark builds a DataSource V2 scan for each subquery before the merge, and each scan holds only the columns of its subquery. So the scans differ, and Spark does not merge them. TPC-DS q9 then reads
store_sales15 times with Vortex, against 5 times with Parquet on the V1 path. Spark's own Parquet V2 reader has the same problem.Spark master fixes this with SPARK-40259 (
TableCapability.SCAN_MERGING), but no Spark release has it.Change
The Vortex session extensions add two rules:
Projectto its own columns.The rules do nothing when a column has different types in the scans (nested schema pruning), or when the widened scans are not equal. They apply only when the session sets
spark.sql.extensions=dev.vortex.spark.VortexSparkSessionExtensions. The user guide has a new "Subquery Merging" section.Results
TPC-DS SF10 q9, local disk,
local[8], Spark 4.2.1:All runs give the same q9 results.
Testing
VortexSubqueryMergeTest:subqueriesThatReadDifferentColumnsShareOneScan: a query in the form of q9 gives 2 scans instead of 4, with the same results.scansThatSparkDoesNotMergeReadOnlyTheirOwnColumns: scans that Spark does not merge read only their own columns.Both tests pass on Spark 3.5 (Scala 2.12 and 2.13), 4.0, 4.1 and 4.2. Both fail without the change.