Repository navigation
Size narrow filtered Scan V2 chunks by fixed-width bytes - #11141
joseph-isaacs wants to merge 1 commit into
Conversation
Filtered scans used 1,048,576-row chunks only when they referenced a single primitive or boolean field, so a filter over two narrow columns fell back to 131,072-row chunks and paid eight times the chunk setup and read fragmentation. Sum the referenced fields' fixed widths instead and keep the large chunks while they total at most 32 bytes per row. Wider and variable-width scans keep the smaller chunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9gdaaoxAeUkYj48Q3 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.039x ➖, 0↑ 2↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.980x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.049x ➖, 0↑ 3↓)
datafusion / parquet / ns (1.037x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 1↑ 2↓)
duckdb / parquet / ns (1.039x ➖, 0↑ 0↓)
|
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Summary
Split out of #10964 as an independently measurable piece.
Filtered scans used 1,048,576-row chunks only when they referenced a single primitive or boolean field. A filter over two narrow columns, such as TPC-H Q6's, therefore fell back to 131,072-row chunks and paid eight times the chunk setup and read fragmentation.
Changes
filtered_rows_per_chunksums the fixed widths of the referenced fields and keeps 1,048,576-row chunks while they total at most 32 bytes per row. Wider and variable-width scans keep 131,072-row chunks.i64columns get the large chunks; five do not.Local validation
4 vCPU cloud VM, TPC-H SF10 Vortex files, Scan V2, 2 alternating (ABBA) rounds × 5 iterations against
develop(246a397). Base-vs-base noise: +1.3% geomean, up to ±8% per query.On four cores, DuckDB gains less from fewer, larger chunks. The original PR measured this change on 16 cores; CI will show whether DuckDB benefits there.
cargo nextest run -p vortex-layout: 737 passed.rustfmt --checkon changed files: clean.cargo clippy: not yet run.Generated by Claude Code