Repository navigation
feat(scan): select the V2 scan from vortex-file, DataFusion and DuckDB - #10551
joseph-isaacs wants to merge 1 commit into
Conversation
- vortex-file: footer open and footer prune stages, `scan_file` to describe a file to the scan, and end-to-end tests of the IO services. - DataFusion and DuckDB select V2 when it is enabled and supported. Files with list columns, and scans combining a filter with a limit, use the default executor. - DuckDB can prepare later files while earlier ones scan (`VORTEX_DUCKDB_FILE_PREFETCH`, off by default); DataFusion reports predicate and IO diagnostics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hai26WGsNERxRHXpA3JboE Signed-off-by: Claude <noreply@anthropic.com>
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.8 µs | 7.6 µs | -11.01% |
| ⚡ ℹ️ | Simulation | test_scan_filter[int] |
3.4 ms | 3 ms | +12.79% |
| ⚡ | WallTime | words_gather_dispatch_neon[65536] |
2.3 µs | 2 µs | +10.22% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/scan-v2-4-wiring (9f5906e) with ji/scan-v2-3-layout (29f589d)
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
Closing as superseded: the V2 scan is wired into DataFusion by #10708 and into DuckDB by #10809, and benchmarks now choose the scan version per benchmark (#11049, #11097). Generated by Claude Code |
Summary
4/5 of the V2 scan stack, stacked on #10550. Split out of #10464.
Stack:
vortex-layout(feat(layout): planner-driven V2 scan over physical plans #10550)vortex-file, DataFusion and DuckDB (this PR)From #10464, on a 4-core VM, DataFusion over Vortex at SF=1, V2 vs the default scan: TPC-H geomean 0.93x, TPC-DS 0.94x.
Changes
scan_file, which describes a file to the scan. With caching on, the file keeps its lowered plans.VORTEX_DUCKDB_FILE_PREFETCH, off by default).Rebase note:
developgained #10486, which moves a DuckDB file's reader and cache into aFileScanStatedropped outside the global multi-file lock. The openedVortexFilethat V2 reads through now lives in that state too, so it gets the same guarantee, and a skipped file (including one skipped during prefetch) drops its state right away.Checks run:
cargo clippy -p vortex-io -p vortex-scan -p vortex-layout -p vortex-file -p vortex-datafusion --all-targets --all-features -- -D warningscargo +nightly-2026-09-10 fmt --checkfor the changed cratescargo test -p vortex-layout -p vortex-scan -p vortex-file --lib, and-p vortex-layout -p vortex-filewithVORTEX_SCAN_V2=1Not run:
vortex-duckdbwas not built, because its build downloads the DuckDB source and that download is blocked in this environment. The DuckDB conflict resolution with #10486 is unchecked by a compiler and needs CI.🤖 Generated with Claude Code
https://claude.ai/code/session_01Hai26WGsNERxRHXpA3JboE
Generated by Claude Code