Repository navigation
Replace layout lowering with a VTable::new_plan method - #11200
Conversation
Each layout now builds its own physical plan through a new `VTable::new_plan` method, mirroring `new_reader`, instead of the central `plan::physical::lower` function that downcast to every known layout. `Layout::new_plan` and `dyn DynLayout::new_plan` expose it. The `lower_*` bodies move unchanged into each layout's `VTable` impl, and `lazy_slots`/`lazy_slots_from` become the crate-private `PlanSlots::from_layout` and `PlanSlots::from_layout_slots`. Layouts without a plan keep the default, which returns the same "No physical plan implementation" error as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBEKjcmqmGZoz9fPBHQ Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.020x ➖, 0↑ 1↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.017x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.038x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 1↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.945x ➖, 3↑ 2↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.931x ➖, 2↑ 0↓)
duckdb / parquet / ns (0.980x ➖, 1↑ 1↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 2↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBEKjcmqmGZoz9fPBHQ Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will improve performance by 10.55%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Summary
Follow-up to #11178. Building a physical plan from a layout was done by one free function,
plan::physical::lower, which downcast the layout to each known type in turn. As a result, the plan for a layout was defined away from the layout itself, and a new layout could only get a plan by editinglower.Now each layout builds its own plan through a
VTable::new_planmethod, the same way it already builds a reader throughnew_reader. The plans produced are unchanged.Changes
New trait method
VTable::new_plan(layout: &Layout<Self>) -> VortexResult<PlanRef>.No physical plan implementation for layout '<id>'error thatlowerreturned. Layouts with no plan, such as indexed, CUDA flat and foreign layouts, behave as before.new_reader:Layout::<V>::new_plan()on a typed layout.DynLayout::dyn_new_plan(), which has a default body becauseForeignLayoutimplementsDynLayoutdirectly.new_plan()on aLayoutRef.Implementations moved, not rewritten
lower_*body moved unchanged into its layout'simpl VTable: Flat, Chunked, Struct, Dict and List.zoned_planhelper. The"Zoned plan requires a zoned layout"branch is gone, because the method now only runs on those two types.lazy_slotsandlazy_slots_frommoved toslots.rsas the crate-privatePlanSlots::from_layoutandPlanSlots::from_layout_slots. They callchild.new_plan()where they used to calllower(&child).Removed
plan/physical/lower.rsand its re-export are deleted.scan/v2and the plan tests now call.new_plan().No tests were added. The existing plan and pipeline tests cover the same plans through the new entry point, including
unsupported_layout_has_no_planfor the default error.API Changes
vortex_layout::plan::physical::lower. Calllayout.new_plan()instead.VTable::new_planandDynLayout::dyn_new_plan. Both have default bodies, so existing implementations still compile.Layout::new_planandnew_planonLayoutRef.Generated by Claude Code