Repository navigation
Run the FineWeb benchmark on the V2 scan - #11055
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBARcDfvVv4UVqkMzmH Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
f489a37 to
929c02f
Compare
Merging this PR will improve performance by 46.46%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing |
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.976x ➖, 1↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.967x ➖, 2↑ 2↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.946x ➖, 3↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.008x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.023x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.019x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.080x ➖, 0↑ 4↓)
|
Follows #11055 (FineWeb). This moves every remaining benchmark except StatPopGen and PolarSignals onto the planner-driven V2 scan. After this PR, those two are the only benchmarks still on V1. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Moves the FineWeb benchmark onto the planner-driven V2 scan for both DataFusion and DuckDB.
The bench orchestrator (
vx-bench) now chooses the scan version per benchmark (#11049), so the change is one entry inBENCHMARK_SCAN_V2. The bench binaries no longer pick a scan per benchmark.Changes
bench-orchestrator/bench_orchestrator/config.py:Benchmark.FINEWEB: True.An earlier version of this PR also changed how DuckDB's V2 driver cuts a file into parts. Its 65,536-row blocks gave the one-file, ~1M-row FineWeb sample only 16 parts, which made DuckDB 1.3–2.5x slower than V1. #11047 has since made DuckDB V2 parts follow the file's layout splits, the same units a V1 scan hands its threads. That replaces the code those commits patched, so they are dropped.
Results
FineWeb NVMe, median ms. V1 is from develop run 41056498175; V2 is from run 41101644140, whose log reports
Vortex scan: V2.DuckDB is at least as fast on every query except Q0, with Q1 and Q2 about 20% faster. DataFusion is faster in total but slower on Q2, Q7 and Q8: Q8 is 9.5 → 12.2 ms and Q2 is 9.6 → 11.5 ms. Each side is a single 5-iteration run.
Checks
uv run --with pytest pytest tests/test_executor.pyinbench-orchestrator: 33 passed. This includes the test that every benchmark has a scan version.uvx ruff check bench-orchestrator/bench_orchestrator/config.py: passes.prpreset). Its jobs are red only because the PR-comment steps have no PR context on a manually started run.To reproduce locally:
🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniBARcDfvVv4UVqkMzmH