Repository navigation
Push down array_sum in DataFusion - #10854
Conversation
|
How did you convince datafusion to do this? |
Merging this PR will not alter performance1 warning
Comparing Footnotes
|
d201266 to
4a1ae34
Compare
4a1ae34 to
86aa3bf
Compare
|
@joseph-isaacs not sure what you mean? This is just scalarfn pushdown |
a106698 to
6e12746
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.045x ➖, 0↑ 1↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.968x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.010x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.986x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 1↑ 1↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.076x ➖, 0↑ 5↓)
datafusion / parquet / ns (0.941x ➖, 2↑ 2↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 1↑ 3↓)
duckdb / parquet / ns (0.924x ➖, 1↑ 0↓)
|
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
3196f13 to
5e526f3
Compare
## Summary Stacked on #10854. DataFusion evaluates `array_transform` above the Vortex scan, and a projection containing a lambda skips projection pushdown entirely (#10215). This converts `array_transform` to Vortex's `list_transform`. ## Changes - `array_transform(list, (x) -> body)` converts to `list_transform` when the lambda has one parameter. - The body is converted in the lambda's own scope, where `$` is the struct of its parameters, as Vortex runs a body: `x` reads `$.x`, which becomes `@x` once the body is converted. In that scope a column or an enclosing lambda's parameter is unsupported, and outside it so is any lambda parameter. - Filters push down too: a conjunct with `array_transform` converts like any other, so DataFusion drops its `FilterExec` and the scan evaluates it. - The #10215 guard that turned off projection pushdown for any lambda is removed: the split is all-or-nothing, so a lambda left for DataFusion always runs over the raw columns. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Summary
The DataFusion expression converter now pushes
array_sum(list_sum) down to Vortex'slist_sum.Changes
array_sum(xs)converts tolist_sum(xs)withinclude_nans. DataFusion plans it asarray_sum(CAST(xs AS List<Float64>)), and Push down list element casts and bool/int/uint casts to Float64 in DataFusion #11161 pushes that cast down, so Vortex adds the sameFloat64elements in the same order.