Repository navigation
bench(statpopgen): run the suite on DataFusion as well as DuckDB - #10212
Conversation
DataFusion indexes lambda parameters past the end of the projection's input columns, but split_projection reshapes the scan output (pushed expressions, one column per unpushable projection) and reassign_expr_columns only remaps Column indices. The lambda variable then pointed at a duplicated list column, so statpopgen Q6 tried to allocate ~137 GB at 1,000 rows, and a bare array_transform in the select list failed to convert. Fall back to reading the referenced columns unchanged when the projection contains a lambda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Run each benchmark's engine_init_sql in datafusion-bench, switch the statpopgen DataFusion session to the DuckDB SQL dialect so the lambda syntax parses, quote the GT column, and filter query 10 through a subquery instead of a SELECT alias. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Give the statpopgen suite the same DataFusion + DuckDB target coverage as the other SQL suites now that its queries run on DataFusion, drop the DuckDB-only coverage sets it alone used, and update the suite docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
8504919 to
fee08c0
Compare
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.975x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.919x ➖, 4↑ 1↓)
duckdb / parquet / ns (0.991x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.801x ➖, 3↑ 1↓)
datafusion / parquet / ns (0.912x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.971x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.109x ➖, 0↑ 2↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.991x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 4↓)
duckdb / parquet / ns (0.995x ➖, 4↑ 2↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (no group data, 0↑ 0↓)
datafusion / parquet / ns (no group data, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.011x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.987x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.016x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.051x ➖, 0↑ 1↓)
File Size Changes (100 files changed, -0.1% overall, 45↑ 55↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.052x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.926x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.904x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.873x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 2↑ 1↓)
No file size changes detected. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Merging this PR will degrade performance by 28.73%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | decode_bitpacked_chunked_avx512[i64, 524288] |
17.8 µs | 31.6 µs | -43.72% |
| ❌ | Simulation | scalar_access[(1024, true, false)] |
788.2 µs | 1,314.1 µs | -40.02% |
| ❌ | WallTime | decode_bitpacked_chunked_avx2[i64, 524288] |
20.5 µs | 34 µs | -39.92% |
| ❌ | WallTime | decode_bitpacked_chunked_neon[i64, 524288] |
27.3 µs | 41.3 µs | -33.78% |
| ❌ | Simulation | scalar_access[(1024, false, false)] |
728.4 µs | 830.5 µs | -12.29% |
| ❌ | Simulation | decompress[alp_for_bp_f64] |
77 µs | 86.8 µs | -11.36% |
| ❌ | Simulation | decompress[datetime_for_bp] |
146.7 µs | 164 µs | -10.53% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/statpopgen-datafusion (c5f6e7c) with ji/datafusion-lambda-projection (012ae44)2
Footnotes
-
509 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
ji/datafusion-lambda-projection(5c4b574) during the generation of this report, so 964ad2c was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ Signed-off-by: Claude <noreply@anthropic.com>
Summary
Stacked on #10215 (Vortex projection-pushdown fix for lambdas); review that first. This PR only contains the statpopgen benchmark changes.
The statpopgen suite only ran on DuckDB because its queries use DuckDB list lambdas. DataFusion 55 already has every function the queries need (
list_sum,list_transformwith lambdas,array_length), so no DataFusion change is needed, only dialect and SQL fixes.Changes
datafusion-benchnow runs each benchmark'sengine_init_sql(Engine::DataFusion)when it creates a session, asduckdb-benchalready does for DuckDB.StatPopGenBenchmark::engine_init_sqlsetsdatafusion.sql_parser.dialect = 'duckdb'for DataFusion, so thelambda g: ...syntax parses (the default generic dialect rejects lambdas).statpopgen.sql: quote"GT"everywhere (DataFusion lowercases unquoted identifiers), rename the lambda parameter fromGTtog, and make query 10 filter a subquery instead of referring to a SELECT alias inWHERE(DuckDB-only behaviour). These are intended to leave DuckDB's plans unchanged; see below.statpopgencatalog entry now uses the standard DataFusion + DuckDB coverage sets. The DuckDB-only sets it alone used are removed, andtest_matrix.pyasserts both engines are present.statpopgen.mdupdated.Results
DataFusion, SF 10 (10,000 variants), median of 5 runs on a 4-core / 15 GB container. All 11 queries pass their expected row counts.
array_lengthGTNot yet measured: DuckDB numbers, and DuckDB before/after the SQL edits. The DuckDB build could not fetch its sources in the environment used, so this PR's CI run is the first DuckDB check.
Checks run:
cargo clippy -p datafusion-bench -p vortex-bench --all-targets --all-features -- -D warnings, nightlycargo fmt --check, bench-orchestratorpytest tests/test_matrix.pyandruff.🤖 Generated with Claude Code
https://claude.ai/code/session_01SzyZPNAEc6NDXftguW3WNQ