Repository navigation
Skip unused fields of nullable structs in V2 scans - #11638
Conversation
Merging this PR will improve performance by 14.23%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
An unused sibling could force the planner to read every field or fail when it could not compile the sibling layout. Preserve ancestor validity in the field projection so the planner can skip unreferenced fields. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
59234fd to
fa6cf9b
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.035x ➖, 0↑ 2↓)
datafusion / vortex-compact / ns (1.032x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.010x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.024x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.984x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.991x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.970x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.022x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.053x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.176x ➖, 1↑ 4↓)
datafusion / parquet / ns (0.994x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (0.884x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.153x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.964x ➖, 0↑ 0↓)
|
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.020x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.015x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.014x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 1↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Nested TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.013x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.988x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: JOB on NVME 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 1↑ 2↓)
duckdb / duckdb / ns (1.002x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 1↓)
File Size Changes (33 files changed, +0.0% overall, 33↑ 0↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.992x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.991x ➖, 1↑ 0↓)
duckdb / duckdb / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.121x ➖, 0↑ 3↓)
datafusion / vortex-compact / ns (1.031x ➖, 1↑ 3↓)
datafusion / parquet / ns (1.070x ➖, 0↑ 4↓)
duckdb / vortex-file-compressed / ns (0.885x ➖, 2↑ 0↓)
duckdb / vortex-compact / ns (0.894x ➖, 3↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 0↑ 0↓)
|
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.917x ➖, 3↑ 0↓)
datafusion / vortex-compact / ns (0.827x ➖, 5↑ 1↓)
datafusion / parquet / ns (0.901x ➖, 2↑ 1↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.991x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.937x ➖, 0↑ 0↓)
|
robert3005
left a comment
There was a problem hiding this comment.
I think this one is correct after @myrrc changes
On
develop, a V2 scan that reads one field from a nullable struct can still plan every child. The scan fails if a sibling uses a layout that V2 cannot execute. This PR plans only the requested fields and the validity needed to preserve their nulls.For
address.zip, the scan can skipaddress.cityand an unused sibling list. A nulladdressstill produces a nullzip.Related to #10952, which handles deep projections in the classic reader.
This change was implemented and reviewed with OpenAI Codex.