Skip to content

do not merge(scan): planning, physical plans, and plan execution - #10339

Closed
joseph-isaacs wants to merge 7 commits into
developfrom
ji/planning-plan-planexec
Closed

joseph-isaacs wants to merge 7 commits into
developfrom
ji/planning-plan-planexec

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Do not merge. This is the whole V2 scan in one place, for review and benchmarking. It is rebuilt on develop's push-based pipelines (#10391–#10393): every physical plan now runs on the pipeline Scan, and this PR's earlier ExecGraph and PipelineGraph executors are removed.

The mergeable pieces are a stack of draft PRs:

  1. feat(layout): zoned pruning, shares, and dictionary value reuse in physical plans #10462: plan. Zoned pruning, Share, Take value rules, Eval on encoded predicates, zoned lowering, per-split ValueCache, WholeSource.
  2. feat(file): a coalescing IO service for planning-protocol scans #10463: IO. The coalescing FileIoService/FileScanIo, read_at changes, read driver, and get_if_ready on segment caches.
  3. feat(scan): planner-driven V2 scans on the physical plan pipelines #10464: planning. The planning driver, footer open/prune stages, announce/pruning/filter/projection planners, the V2 ScanBuilder, and DataFusion/DuckDB selection.

This branch is those three commits plus one do-not-merge commit holding:

Previous head: 6115639.

Changes

  • Plan vs planning. Plans compute the rows of a range under a mask, compiled to pipelines with the mask known at compile time. Planning drives them per split: announce reads, prune with zone statistics, run filter conjuncts one at a time (adaptive order, each narrowing the mask), then project the surviving rows. Each stage is one pipeline Scan.
  • Caching is bounded. Dictionary and shared values live in a per-split ValueCache that is dropped with the split. Zone tables live for the query. Nothing decoded is kept across scans.
  • Fallbacks. Files with list columns, and scans combining a filter with a limit, use the default executor.

Local results on a 4-core VM, DataFusion over Vortex at SF=1, V2/V1 geomean of per-query medians:

Suite Geomean V2/V1
TPC-H 0.93x
TPC-DS 0.94x

No reproducible regressions when the apparent losses were rerun with alternating V1/V2 runs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature do not merge Pull requests that are not intended to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants