Skip to content

Spark: let Spark merge subqueries that scan one Vortex table - #11330

Draft
robert3005 wants to merge 11 commits into
rk/sparkfileformat-pushdownsfrom
rk/v2-merge-scalar-subqueries
Draft

robert3005 wants to merge 11 commits into
rk/sparkfileformat-pushdownsfrom
rk/v2-merge-scalar-subqueries

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Stacked on #11096.

Problem

Spark merges scalar subqueries that have the same filters into one scan. It does this only when their scans are equal. Spark builds a DataSource V2 scan for each subquery before the merge, and each scan holds only the columns of its subquery. So the scans differ, and Spark does not merge them. TPC-DS q9 then reads store_sales 15 times with Vortex, against 5 times with Parquet on the V1 path. Spark's own Parquet V2 reader has the same problem.

Spark master fixes this with SPARK-40259 (TableCapability.SCAN_MERGING), but no Spark release has it.

Change

The Vortex session extensions add two rules:

  • Widen runs before Spark merges subqueries. It finds the Vortex scans that differ only in their columns (same files, options and pushed filters). It changes each to read the union of their columns, below a Project to its own columns.
  • Narrow runs after the merge. It changes each scan that Spark did not merge back to its own columns, so those scans read no extra data.

The rules do nothing when a column has different types in the scans (nested schema pruning), or when the widened scans are not equal. They apply only when the session sets spark.sql.extensions=dev.vortex.spark.VortexSparkSessionExtensions. The user guide has a new "Subquery Merging" section.

Results

TPC-DS SF10 q9, local disk, local[8], Spark 4.2.1:

Scans Bytes read Median time
Vortex before 15 1031 MB 518 ms
Vortex after 5 764 MB 288 ms
Parquet V1 5 98 MB 730 ms
Parquet V2 15 295 MB 1186 ms

All runs give the same q9 results.

Testing

VortexSubqueryMergeTest:

  • subqueriesThatReadDifferentColumnsShareOneScan: a query in the form of q9 gives 2 scans instead of 4, with the same results.
  • scansThatSparkDoesNotMergeReadOnlyTheirOwnColumns: scans that Spark does not merge read only their own columns.

Both tests pass on Spark 3.5 (Scala 2.12 and 2.13), 4.0, 4.1 and 4.2. Both fail without the change.

robert3005 and others added 11 commits November 8, 2026 00:27
Reads now hand Spark's pushdowns to the native scan wherever Vortex can
answer them:

- Aggregates: COUNT(*), COUNT, MIN, MAX, and integer SUM (so AVG), grouped
  by any partition columns, answered from footer statistics or computed
  natively from only the columns involved (new NativeFiles.aggregate).
  Footer-only aggregates are answered on the Spark driver.
- Runtime filters: dynamic pruning on partition columns drops files on the
  driver; on data columns readers evaluate the keys as a native set. An
  optimizer rule stops pruning subqueries from nesting through Vortex
  scans, which made TPC-DS q64 plan 122,883 subqueries.
- LIMIT, OFFSET (whole files skipped through footer row counts),
  ORDER BY ... LIMIT (a shared, footer-seeded dynamic threshold through the
  new DynamicValue binding), and TABLESAMPLE as a row selection.
- IN as a native set, LIKE patterns and column comparisons with no V1 form,
  nested struct pruning, and partition keys reported for v2 bucketing.
- File splitting into row ranges, _metadata.row_index on the V1 path,
  footer column statistics for the cost-based optimizer, and sorted files
  reported to Spark.

Other changes:

- Spark 4.1 and 4.2 connector builds with their pushdowns.
- Read and write Spark VARIANT columns as Vortex variants, and extract
  variant paths natively.
- Record IsSorted only under editions that include it.
- Send decimals of at most 18 digits to Spark as unscaled longs.
- Scan metrics.

SparkPushdownBenchmark measures each pushdown on and off.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ines

A column that the dictionary layout declines, such as a decimal with many
distinct values, can still have parts with few distinct values, such as its
integer bytes. The fallback path now uses the compressor with IntDictScheme.
For TPC-H SF10 l_quantity this reduces the column from 97.6 MB to 45.2 MB.

Signed-off-by: Robert Kruszewski <github@robertk.io>
A Cleaner frees the native pointer of an Expression when the Expression
becomes unreachable. The factories read the pointers of their inputs and then
call native code. After the pointer reads the JIT can treat the inputs as
unreachable, so a garbage collection during the call could free an input that
native code still reads. A Spark TPC-DS run through Iceberg crashed with
SIGSEGV in NativeExpression.literalList for a runtime-filter IN list.

The factories, DynamicValue and DataSource.scan now keep their inputs
reachable with Reference.reachabilityFence until the native call returns.

Signed-off-by: Robert Kruszewski <github@robertk.io>
With 64 MiB splits, scans of tables of about 1 GB ran fewer tasks than there
are cores. With 16 MiB on Spark 4.2, SF10 and 32 cores, TPC-H goes from 38.96 s
to 36.3 s and TPC-DS from 98.4 s to 85.5 s. ClickBench does not change
(52.72 s and 52.75 s).

Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark sizes splits by an even share of the bytes per core. Vortex cuts files
only at chunk boundaries, and packing cannot fill each split, so a file slightly
larger than the share became two pieces and the scan took two waves of tasks.
With 56 MB files of store_sales on 32 cores, a scan ran about 38 tasks. The
share now gets a third more room.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Split planning read the layouts of all the columns of each file. For a
scan of a few columns of a wide file, such as ClickBench hits with 105
columns, this used about 9% of the query time. VortexFile::splits_for
and NativeFiles.splits now take the scanned columns.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Each task converts the runtime join filter of its scan. The filter can
hold thousands of values, and one Java expression and native literal per
value cost about 80 ms per task in TPC-DS q90. Integer and string lists
now go to native code as one array.

Signed-off-by: Robert Kruszewski <github@robertk.io>
On Java 17, Spark joins the partition files of a shuffle task with a
memory map and unmap per file. The unmaps lock the address space of the
executor, and concurrent Vortex scans wait on them. TPC-DS q31 on 32
cores took 1.15 s on Java 17 and 0.65 s on Java 21.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark 4 enables ANSI rules by default, and the scan kept every SUM in
Spark under them, because a wrapped 64-bit per-file sum cannot tell an
overflow. A sum of TINYINT, SMALLINT, or INT values cannot overflow in a
file of up to 2^32 rows, so these sums are now pushed under ANSI rules.
A file with more rows fails the query with a request to disable
aggregate pushdown. The driver merges footer partials with exact
addition under ANSI rules.

Spark states SUM(col + k) to a source only under ANSI rules. The scan
answers it as SUM(col) + k * COUNT(col) when no value of the column plus
the constant can overflow the type of the addition: a TINYINT or
SMALLINT column plus an INT or BIGINT constant, or an INT column plus a
BIGINT constant. ClickBench q30 sums SMALLINT ResolutionWidth plus 0 to
89.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark merges scalar subqueries with the same filters into one scan only
when their scans are equal. A DataSource V2 scan holds only the columns
of its subquery, so the scans of TPC-DS q9 differ and Spark reads
store_sales 15 times. Spark's own Parquet V2 reader has the same problem.

The session extensions now add two rules. Before Spark merges subqueries,
Widen changes the Vortex scans that differ only in their columns to read
the union of their columns, below a projection to their own columns.
After the merge, Narrow changes each scan that Spark did not merge back
to its own columns.

On TPC-DS SF10 q9 the Vortex scans go from 15 to 5, as on the V1 path.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 force-pushed the rk/sparkfileformat-pushdowns branch 4 times, most recently from 4a7cf7e to 9f922e9 Compare November 12, 2026 10:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant