Repository navigation
feat(layout): planner-driven V2 scan over physical plans - #10550
joseph-isaacs wants to merge 1 commit into
Conversation
An opt-in scan (`VORTEX_SCAN_V2=1`) that lowers a layout once into a physical plan and runs each split through announce, pruning, filter (adaptive conjunct order) and projection planners. Each stage of a split runs as one pipeline `Scan`, and a split's scans share a `ValueCache` dropped with the split. Adds the V2 `ScanBuilder`/`RepeatedScanV2`, plan-derived splits, early IO announcements and file pruning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hai26WGsNERxRHXpA3JboE Signed-off-by: Claude <noreply@anthropic.com>
Merging this PR will improve performance by 15.28%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | threshold_neon[80] |
4.5 µs | 3.8 µs | +16.01% |
| ⚡ ℹ️ | Simulation | test_scan_scalar_at[int] |
896.1 µs | 782.2 µs | +14.56% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/scan-v2-3-layout (29f589d) with ji/scan-v2-2-driver (da97422)
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
Closing as superseded: the Generated by Claude Code |
Summary
3/5 of the V2 scan stack, stacked on #10549. Split out of #10464.
Stack:
vortex-layout(this PR)vortex-file, DataFusion and DuckDBAdds an opt-in scan (
VORTEX_SCAN_V2=1) that lowers a file's layout once into a physical plan and runs each split through planners on the planning protocol, executing every plan on develop's pipelineScan. No engine selects it until the next PR.Changes
Only
vortex-layoutchanges.scan/planning): announce, pruning, filter and projection planners. The filter planner orders conjuncts adaptively.scan/v2): the V2ScanBuilderandRepeatedScanV2, plan-derived splits, early IO announcements and file pruning.Scan.ValueCachethat is dropped with the split, so dictionary values are never kept for the rest of a scan.Shares that no plan reads whole are removed, so pushed-down expressions run over encoded values.ScanBuilder::into_partslets the V2 builder copy the default builder's configuration.Changes needed to rebase onto
developsince #10464:LegacyStatsPlanfor legacyvortex.statslayouts, which the optimizer rewrites intoZoned, matching the merged feat(layout): zoned pruning, shares, and dictionary value reuse in physical plans #10462.share.rskeys plans byPlanByPtrin place of the removedPlanRef::addr.Tests:
scan/v2/tests.rs(including legacy-stats and list cases) andscan/planning/filter/tests.rs.Checks run:
cargo clippy -p vortex-io -p vortex-scan -p vortex-layout -p vortex-file --all-targets --all-features -- -D warningson this commitcargo test -p vortex-layout --lib, with and withoutVORTEX_SCAN_V2=1(on the top of the stack)🤖 Generated with Claude Code
https://claude.ai/code/session_01Hai26WGsNERxRHXpA3JboE
Generated by Claude Code