Repository navigation
Push slices through filtered arrays - #10959
Conversation
Translate filtered output ranks to a bounded child range before slicing, so independent slices do not each execute the entire filtered array. Signed-off-by: Codex <codex@openai.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.005x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.017x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.927x ➖, 3↑ 1↓)
datafusion / parquet / ns (0.926x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.052x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.044x ➖, 0↑ 1↓)
|
Merging this PR will regress 5 benchmarks3 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes |
Slicing a filtered array can execute the entire filtered child before applying the slice. Separate slices can therefore repeat filtering and decoding.
Add a
SliceReducerule forFilterthat translates output ranks through the existing mask, slices the child and mask to the corresponding input range, and then filters that smaller range. The output ranks are remapped; the slice bounds cannot simply be reused against the unfiltered child.Includes ten regression cases covering bounded child spans, nullable values, cached and uncached mask indices, empty/full slices, and single-row slices. Updates the affected execution-trace snapshots with unchanged expected values.
This is the standalone array change from #10799, based on develop
891c01ab40. It changes only four files invortex-array.Validation:
git diff --checkpassed.The earlier V2 ClickBench improvements measured within #10799 are separate from these standalone results.