Repository navigation
bench: run every DataFusion benchmark on the V2 scan - #10977
joseph-isaacs wants to merge 1 commit into
Conversation
The orchestrator now passes --scan-v2 true to datafusion-bench, so every benchmark suite uses the planner-driven V2 Vortex scan rather than only TPC-H, TPC-DS and ClickBench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBD1TEu6bod84sjRqAX Signed-off-by: Claude <noreply@anthropic.com>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Merging this PR will improve performance by 10.49%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.960x ➖, 3↑ 2↓)
datafusion / vortex-compact / ns (1.510x ❌, 1↑ 8↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.948x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (1.195x ❌, 0↑ 4↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.020x ➖, 0↑ 1↓)
File Size Changes (200 files changed, +0.0% overall, 106↑ 94↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.009x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.978x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.772x ❌, 0↑ 7↓)
No file size changes detected. |
Benchmarks: JOB on NVME 📖Commits: PR No baseline is available for this benchmark yet; PR measurements are shown without comparison. duckdb / vortex-file-compressed / ns (no group data, 0↑ 0↓)
duckdb / parquet / ns (no group data, 0↑ 0↓)
duckdb / duckdb / ns (no group data, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.983x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.001x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.994x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.993x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.028x ➖, 1↑ 5↓)
datafusion / vortex-compact / ns (1.153x ❌, 0↑ 18↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.999x ➖, 0↑ 0↓)
vortex / parquet / ns (0.996x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.995x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 1↑ 3↓)
datafusion / vortex-compact / ns (1.020x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.092x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed / ns (0.946x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.020x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.974x ➖, 0↑ 1↓)
|
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.940x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.008x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.792x ➖, 3↑ 0↓)
datafusion / vortex-compact / ns (0.891x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.893x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.050x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.899x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.969x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.812x ➖, 3↑ 0↓)
datafusion / vortex-compact / ns (0.877x ➖, 3↑ 0↓)
datafusion / parquet / ns (0.956x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.920x ➖, 2↑ 0↓)
duckdb / vortex-compact / ns (0.860x ➖, 3↑ 0↓)
duckdb / parquet / ns (0.952x ➖, 1↑ 0↓)
|
Benchmarks: Random Access (S3) 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (0.868x ➖, 2↑ 1↓)
random-access / parquet / ns (0.894x ➖, 3↑ 0↓)
random-access / lance / ns (0.904x ➖, 0↑ 0↓)
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (1.042x ➖, 1↑ 2↓)
random-access / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
random-access / parquet / ns (1.001x ➖, 0↑ 0↓)
random-access / lance / ns (0.986x ➖, 0↑ 0↓)
random-access / litdata / ns (0.998x ➖, 0↑ 0↓)
|
|
Closing as superseded: #11049, #11061 and #11097 now choose the V2 scan per benchmark in the bench orchestrator. V1 is kept only through the Generated by Claude Code |
Summary
Until now,
datafusion-benchused the planner-driven V2 Vortex scan by default only for TPC-H, TPC-DS and ClickBench. This PR switches every DataFusion benchmark suite to V2 by changing the Python bench orchestrator (vx-bench), which runs the benchmarks both in CI and locally.Changes
BenchmarkExecutor.build_commandnow passes--scan-v2 trueto everydatafusion-benchinvocation. DuckDB invocations are unchanged.Checks run:
ruff format --check,ruff check, andpytest bench-orchestrator/tests/test_executor.py(8 passed).Generated by Claude Code