Skip to content

Push every Vortex-capable operation into the Spark file format - #11096

Draft
robert3005 wants to merge 9 commits into
developfrom
rk/sparkfileformat-pushdowns
Draft

robert3005 wants to merge 9 commits into
developfrom
rk/sparkfileformat-pushdowns

Conversation

@robert3005

@robert3005 robert3005 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Push every operation that Vortex can answer into the Spark file format scan.

Changes

  • Aggregates: COUNT, MIN, MAX, integer SUM, grouped by partition columns, from footer statistics or native compute.
  • Runtime filters on partition and data columns. A new rule stops nested pruning subqueries (TPC-DS q64 planned 122,883).
  • LIMIT, OFFSET, ORDER BY ... LIMIT, TABLESAMPLE, IN, LIKE, nested struct pruning.
  • File splitting, _metadata.row_index, footer statistics for the CBO, sorted file reporting.
  • Spark 4.1 and 4.2 builds, and VARIANT read and write.
  • SparkPushdownBenchmark measures each pushdown on and off.

API Changes

  • New Java types: DynamicValue, FileAggregate, FileAggregateResult, NativeFiles.aggregate.
  • New overload Partition.scanArrow(BufferAllocator, boolean).

robert3005 and others added 9 commits November 12, 2026 10:32
Reads now hand Spark's pushdowns to the native scan wherever Vortex can
answer them:

- Aggregates: COUNT(*), COUNT, MIN, MAX, and integer SUM (so AVG), grouped
  by any partition columns, answered from footer statistics or computed
  natively from only the columns involved (new NativeFiles.aggregate).
  Footer-only aggregates are answered on the Spark driver.
- Runtime filters: dynamic pruning on partition columns drops files on the
  driver; on data columns readers evaluate the keys as a native set. An
  optimizer rule stops pruning subqueries from nesting through Vortex
  scans, which made TPC-DS q64 plan 122,883 subqueries.
- LIMIT, OFFSET (whole files skipped through footer row counts),
  ORDER BY ... LIMIT (a shared, footer-seeded dynamic threshold through the
  new DynamicValue binding), and TABLESAMPLE as a row selection.
- IN as a native set, LIKE patterns and column comparisons with no V1 form,
  nested struct pruning, and partition keys reported for v2 bucketing.
- File splitting into row ranges, _metadata.row_index on the V1 path,
  footer column statistics for the cost-based optimizer, and sorted files
  reported to Spark.

Other changes:

- Spark 4.1 and 4.2 connector builds with their pushdowns.
- Read and write Spark VARIANT columns as Vortex variants, and extract
  variant paths natively.
- Record IsSorted only under editions that include it.
- Send decimals of at most 18 digits to Spark as unscaled longs.
- Scan metrics.

SparkPushdownBenchmark measures each pushdown on and off.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Cleaner frees the native pointer of an Expression when the Expression
becomes unreachable. The factories read the pointers of their inputs and then
call native code. After the pointer reads the JIT can treat the inputs as
unreachable, so a garbage collection during the call could free an input that
native code still reads. A Spark TPC-DS run through Iceberg crashed with
SIGSEGV in NativeExpression.literalList for a runtime-filter IN list.

The factories, DynamicValue and DataSource.scan now keep their inputs
reachable with Reference.reachabilityFence until the native call returns.

Signed-off-by: Robert Kruszewski <github@robertk.io>
With 64 MiB splits, scans of tables of about 1 GB ran fewer tasks than there
are cores. With 16 MiB on Spark 4.2, SF10 and 32 cores, TPC-H goes from 38.96 s
to 36.3 s and TPC-DS from 98.4 s to 85.5 s. ClickBench does not change
(52.72 s and 52.75 s).

Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark sizes splits by an even share of the bytes per core. Vortex cuts files
only at chunk boundaries, and packing cannot fill each split, so a file slightly
larger than the share became two pieces and the scan took two waves of tasks.
With 56 MB files of store_sales on 32 cores, a scan ran about 38 tasks. The
share now gets a third more room.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Split planning read the layouts of all the columns of each file. For a
scan of a few columns of a wide file, such as ClickBench hits with 105
columns, this used about 9% of the query time. VortexFile::splits_for
and NativeFiles.splits now take the scanned columns.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Each task converts the runtime join filter of its scan. The filter can
hold thousands of values, and one Java expression and native literal per
value cost about 80 ms per task in TPC-DS q90. Integer and string lists
now go to native code as one array.

Signed-off-by: Robert Kruszewski <github@robertk.io>
On Java 17, Spark joins the partition files of a shuffle task with a
memory map and unmap per file. The unmaps lock the address space of the
executor, and concurrent Vortex scans wait on them. TPC-DS q31 on 32
cores took 1.15 s on Java 17 and 0.65 s on Java 21.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Spark 4 enables ANSI rules by default, and the scan kept every SUM in
Spark under them, because a wrapped 64-bit per-file sum cannot tell an
overflow. A sum of TINYINT, SMALLINT, or INT values cannot overflow in a
file of up to 2^32 rows, so these sums are now pushed under ANSI rules.
A file with more rows fails the query with a request to disable
aggregate pushdown. The driver merges footer partials with exact
addition under ANSI rules.

Spark states SUM(col + k) to a source only under ANSI rules. The scan
answers it as SUM(col) + k * COUNT(col) when no value of the column plus
the constant can overflow the type of the addition: a TINYINT or
SMALLINT column plus an INT or BIGINT constant, or an INT column plus a
BIGINT constant. ClickBench q30 sums SMALLINT ResolutionWidth plus 0 to
89.

Signed-off-by: Robert Kruszewski <github@robertk.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant