Repository navigation
feat: push down list_sum into Vortex scans in DataFusion and DuckDB - #10234
joseph-isaacs wants to merge 13 commits into
Conversation
DataFusion coerces array_sum's argument to a list of Float64 and sums it in f64, skipping null elements and returning null for null, empty and all-null lists, with NaN poisoning the sum. Convert it to Vortex list_sum with NaNs included, over the converted cast, so the sum runs in the scan instead of after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
DuckDB binds list_sum(col) as list_aggr(col, 'sum'), which the scalar function pushdown never offered because it only accepted single-argument functions. Accept functions whose extra arguments are constants, and convert list_aggr/list_aggregate/array_aggr(col, 'sum') over DOUBLE[] to Vortex list_sum with NaNs included. Integer lists sum into HUGEINT, which Vortex can't produce, so they stay in DuckDB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Add two lambda-free list_sum queries, per-variant and over the whole dataset, so the suite measures list aggregation on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
|
Generated by Claude Code |
Merging this PR will improve performance by 26.23%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | filter_powerlaw_by_mostly_true[250000] |
150.9 µs | 104.7 µs | +44.06% |
| ⚡ | WallTime | compare_u64_avx2 |
4.1 µs | 3.7 µs | +10.61% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/list-sum-pushdown (6375b8d) with develop (9a08b82)
Footnotes
-
503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
The Generated by Claude Code |
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.970x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.048x ➖, 2↑ 2↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.981x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.977x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.017x ➖, 0↑ 1↓)
File Size Changes (100 files changed, +0.1% overall, 57↑ 43↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.057x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.940x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.934x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.981x ➖, 3↑ 0↓)
duckdb / parquet / ns (0.981x ➖, 3↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 5↑ 3↓)
duckdb / parquet / ns (0.996x ➖, 5↑ 4↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.987x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 2↑ 3↓)
datafusion / parquet / ns (1.086x ➖, 0↑ 4↓)
duckdb / vortex-file-compressed / ns (0.948x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.023x ➖, 0↑ 3↓)
|
list_map applies an element expression to every list element and keeps the list structure, so engines can push down lambdas such as array_transform(xs, x -> x IS NOT NULL). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Summing a list of booleans fell back to slicing every list and running a fresh accumulator per row. Count each list's valid true bits instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
…uments into Vortex scans - Convert array_transform lambdas over the list element to Vortex list_map. - Push the convertible sub-expressions of projections that can't be pushed whole, instead of disabling pushdown for any projection with a lambda. - Add VortexExpressionPushdown, a physical optimizer rule that merges projections over Vortex scans and moves aggregate arguments such as SUM(array_sum(xs)) into the scan. - Show the projection the Vortex scan evaluates in EXPLAIN. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Summing cast(ints as f64) per list materialized every cast element before summing. Convert each element inside the summation loop instead, with the same per-element conversion and order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
…nts, keep lambda filters in DataFusion The aggregate rewrite passed every input column through the scan, so SUM(array_sum(GT)) still decoded GT. Keep only the columns the aggregate reads. Evaluating lambda predicates in the scan made statpopgen query 10 twice as slow, so leave them to DataFusion's filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
…s expressions Printing it for every scan changed EXPLAIN output for scans that only read columns. Update decimal_to_integer.slt, whose EXPLAIN now shows the cast running inside the Vortex scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
A row function took each input's validity before decoding it. For a lazy scalar function input such as list_sum, that validity recomputed the whole function when the output was masked, so 1 - a / (2 * b) over two list sums ran each sum twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Summing lists over encoded elements decoded the whole element array first, which for bit-packed lists wrote far more memory than the sum read. Decode cache-sized slices instead and carry each list's running sum across them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
|
Generated by Claude Code |
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
Summary
Vortex already has a
list_sumscalar function (vortex.list.sum), but neither engine integration pushed list sums into the scan, so they always ran after the full lists were decoded. Follows #10215 and #10212, which got statpopgen running on DataFusion.Changes
DataFusion (
vortex-datafusion/src/convert/exprs.rs)array_sum(x)(aliaslist_sum) reaches the scan asarray_sum(CAST(x AS List(Float64))). It now converts tocast(vortex.list.sum(cast(x as list(f64?)), skip_nans=false) as f64?).test_array_sum_pushed_down_as_list_sumchecks the conversion;test_array_sum_pushdownruns the same query with projection pushdown off and on, asserts identical results, and snapshots them.DuckDB (
vortex-duckdb/cpp/scalar_fn_pushdown.cpp,vortex-duckdb/src/convert/expr.rs)list_sum(col)aslist_aggr(col, 'sum'). The scalar-function pushdown only offered single-argument functions, so it now also offers functions whose extra arguments are constants. Anything the Rust converter doesn't recognise is rejected and treated as a conflict, as before.list_aggr/list_aggregate/array_aggr/array_aggregatewith'sum'overDOUBLE[]convert to Vortexlist_sumwith NaNs included.HUGEINT, which Vortex can't produce.FLOAT[]is cast toDOUBLE[]by DuckDB before the call, so it isn't a bare column.vortex-sqllogictest/slt/duckdb/list_sum_pushdown.slt(expected values taken from DuckDB 1.5.5).Benchmark (
vortex-bench/sql/statpopgen.sql)SELECT "CHROM", "POS", LIST_SUM("GT")and q12SELECT SUM(LIST_SUM("GT")).Local results
DataFusion, SF 10, Vortex, warm runs alternated between builds over 3 rounds (median, p25–p75, ms):
list_sumpushdownLIST_SUMSUM(LIST_SUM)q12 is unchanged as expected: DataFusion keeps
list_suminside the aggregate's arguments and doesn't offer it to the scan. statpopgen'sGTisUBIGINT[], so DuckDB doesn't push these queries down.Checks
cargo test -p vortex-datafusion --lib(336 passed),cargo clippy -p vortex-datafusion -p vortex-bench --all-targets --all-features -- -D warnings, nightlycargo fmt --check.datafusion-benchstatpopgen q9–q12 row counts at SF 1 and SF 10, Parquet and Vortex.clang-formaton the C++ change.vortex-duckdb(its build couldn't fetch DuckDB sources in the environment used), so the DuckDB Rust/C++ changes and the.sltare first compiled and run by this PR's CI.🤖 Generated with Claude Code
https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ
Generated by Claude Code