Repository navigation
feat(scan): planner-driven V2 executor for projection scans behind one fork - #10530
joseph-isaacs wants to merge 8 commits into
Conversation
Describe how to land the vortex-scan planning driver running develop's pipeline::Scan decode path, with IO through the planning protocol, as a small PR stack extracted from 42b29cc. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P1oQtzTgSzkvrGDKisoNko Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Claude <noreply@anthropic.com>
…e fork Port the planning driver (`Run`/`Driver`) into vortex-scan and a projection-only V2 executor into vortex-layout that runs develop's pipeline `Scan` through the planning protocol, with every segment read issued as an `IoRequest` served from the file's `SegmentSource`. `ScanBuilder`'s lazy prepare step is the single place a streaming scan picks its executor: `v2::select` returns V1 unless `VORTEX_SCAN_V2=1`, the builder carries a `ScanFile`, and the scan has no filter, no limit and no list layout. `VortexFile::scan()` attaches the `ScanFile`; no caller changes. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6
…ndoff doc Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6
Develop's lower keeps Zoned and LegacyStats nodes in the plan, so the boundary walk stopped at them and every default-written file ran as a single split. Follow their data child. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6
Each split had its own ValueCache, so a dictionary shared by many chunks was decoded once per split (HashTags 2.2x, CMSprovider 1.7x slower than V1). Share one cache per scan, as V1's reader does. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6
Merging this PR will regress 4 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ |
Simulation | test_repeated_scan_scalar_at[int_str_float] |
539 µs | 628.6 µs | -14.24% |
| ❌ ℹ️ | Simulation | test_scan_filter_to_arrow_array[int_float] |
3.5 ms | 4.1 ms | -13.72% |
| ❌ ℹ️ | Simulation | test_repeated_scan_filter[int_float] |
3.4 ms | 3.9 ms | -11.17% |
| ❌ ℹ️ | Simulation | test_repeated_scan_filter[int] |
2.5 ms | 2.7 ms | -10.44% |
| ⚡ ℹ️ | Simulation | test_repeated_scan_scalar_at[int_float] |
486.7 µs | 393.3 µs | +23.77% |
| ⚡ | WallTime | bitpack_blocked_compress_avx512 |
5.6 µs | 4.8 µs | +18.01% |
| ⚡ | WallTime | bitpack_blocked_compress_avx2 |
7.6 µs | 6.7 µs | +13.08% |
| ⚡ | WallTime | filter_neon[80] |
2.9 µs | 2.6 µs | +12.48% |
| ⚡ | WallTime | words_gather_dispatch_avx2[65536] |
1.4 µs | 1.3 µs | +10.27% |
| 🆕 | Simulation | prepare_and_compile[1] |
N/A | 52.7 µs | N/A |
| 🆕 | Simulation | prepare_and_compile[100] |
N/A | 322.3 µs | N/A |
| 🆕 | Simulation | prepare_and_compile[1000] |
N/A | 2.6 ms | N/A |
| 🆕 | Simulation | prepare[1] |
N/A | 16.3 µs | N/A |
| 🆕 | Simulation | prepare[100] |
N/A | 74 µs | N/A |
| 🆕 | Simulation | prepare[1000] |
N/A | 424 µs | N/A |
| 🆕 | Simulation | scan[1] |
N/A | 96.6 µs | N/A |
| 🆕 | Simulation | scan[100] |
N/A | 1.3 ms | N/A |
| 🆕 | Simulation | scan[1000] |
N/A | 12.3 ms | N/A |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/planner-decode-slice-migration-xppbf1 (7b15cfc) with develop (18aebea)
Reuse V1 split policies, avoid redundant identity-projection planning, and release decoded dictionary values after their consuming splits finish. Reduce pipeline allocation, plan retention, and ready-I/O scheduling overhead. Validated with repeated V1/V2/Parquet decompression comparisons, output parity across 14 datasets, and 175 focused pipeline and scan tests. Some workloads still retain modest RSS and parallel duplicate-read overhead versus V1. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Measure preparation, compilation and in-memory segment delivery for narrow and wide pipeline scans in the existing CodSpeed layout shard. Document the stable identity of scoped dictionary-cache keys and use an explicit Arc clone in the V2 test so the scan code passes the CI Clippy configurations. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Keep private implementation names as code and remove the redundant v2 link target so the planner documentation builds with warnings denied. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.990x ➖, 1↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.991x ➖, 1↑ 0↓)
vortex / parquet / ns (1.004x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (1.019x ➖, 0↑ 5↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Summary
First mergeable slice of the V1 to V2 scan migration, following the handoff in
vortex-scan/design/planner-decode-slice.md(#10464 / #10466 trimmed down). Projection-only scans can run on develop's pipelineScan, driven by thevortex-scanplanning protocol: every segment read is anIoRequestserved from the file'sSegmentSource.V1 and V2 fork in one place only. The source commit put
if v2::enabled() && v2::supports_scan(..) { v2::into_stream(..) } else { b.into_stream() }at every caller. Here,ScanBuilder's lazy prepare step is the only fork:v2::selectowns the gate:VORTEX_SCAN_V2=1, the builder carries aScanFile, there is no filter or limit, and the file has no List layouts (lists written as flat leaves, the default, are supported). Each later PR widens this one gate.into_stream,into_iterand the Arrow record-batch stream all go through it, so no caller changes. The decision is logged atdebugon targetvortex_layout::scan::v2.Changes
vortex-scan: the planning driver (Run/Driver) from feat(scan): planner-driven V2 scans on the physical plan pipelines #10464, with tracing stripped so it does not depend on feat(file): a coalescing IO service for planning-protocol scans #10463, plus its scripted tests.vortex-layout:scan/planning/:ProtocolGraphmaps pipelineTurn::ReadtoFetchrequests.ProjectionPlanner/ProjectionMorselwith prefetch and forget dropped.scan/v2/:select) andPreparedScan.SegmentScanIoadapts aSegmentSourcetoIoService.SplitPlanand the chunk-start splitter.ScanBuilder::with_scan_file.vortex-file:VortexFile::scan_file(), andVortexFile::scan()attaches it.lowerkeepsZoned/LegacyStatsplan nodes, so the boundary walk now follows their data child. Before this, every default-written file ran as a single split.scan/v2/tests.rs):row_idxprojection.ScanFileand List-layout scans stay on V1.Local verification
cargo clippy -p vortex-scan -p vortex-layout -p vortex-file --all-targets --all-features -- -D warnings: clean.cargo +nightly-2026-09-10 fmton those crates.cargo test -p vortex-layout --lib: 530 passed.cargo test -p vortex-scan --lib: 59 passed.compress-bench --formats vortex --ops decompresswithVORTEX_SCAN_V2=1: all 15 datasets ran on V2 (15scan runs on v2, 0 fallbacks).vortex-filetests, and the DataFusion / DuckDB integrations (they build their ownScanBuilderand stay on V1 until they callwith_scan_file).Local decompression times, 4-core container, 5 iterations, V1 (
VORTEX_SCAN_V2=0) vs V2 (=1):Geomean V2/V1: 1.00x. These are single local runs on a noisy shared container. The design doc's open risk (the full stack regressed on CI's 96-core runners) still needs a CI benchmark run to rule out.
API Changes
All additive. V1 behaviour is unchanged unless
VORTEX_SCAN_V2=1is set:ScanBuilder::with_scan_fileVortexFile::scan_filevortex_layout::scan::v2::{ScanFile, SegmentLocation, ENV_VAR, enabled}vortex_scan::planning::driver🤖 Generated with Claude Code
https://claude.ai/code/session_017cUKVPYqfN7SLcXetK59i6
Generated by Claude Code