Repository navigation
do not merge(scan): V2 planning, local IO coalescing, and shared segment cache on top of #10339 - #10349
Closed
joseph-isaacs wants to merge 96 commits into
Closed
do not merge(scan): V2 planning, local IO coalescing, and shared segment cache on top of #10339#10349joseph-isaacs wants to merge 96 commits into
joseph-isaacs wants to merge 96 commits into
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ji/planning-plan-planexec)Benchmark-only draft: this branch is #10339 plus three commits, opened against
developso theaction/bench-sqllabel can run the SQL benchmarks (which already setVORTEX_SCAN_V2=1on that branch). Only the three commits below are new relative to #10339.Changes
perf(scan): avoid whole-struct expansion when pushing expressions through Pack(vortex-layout/src/plan/plans/pack.rs):ExpressionPackRuleexpands the struct root lazily (only the field accesses an expression uses), annotates free fields once, and checks identity expressions structurally. Trims V2 per-file planning on wide tables such as ClickBenchhits.perf(io): coalesce local reads across at most 64 KiB gaps(vortex-io):ObjectStoreReadAtover object_store'sLocalFileSystemuses the local-file coalesce config instead of the object-storage one, and that config's gap shrinks from 1 MiB to 64 KiB (4 MiB cap unchanged). Remote stores are unchanged. With V2 announcing every segment of a split up front, the old gap let the coalescer chain gaps (ClickBench q20 read 430 MB to serve 277 MB of requested segments). Adds a unit test for the local-store detection.perf(scan): share decoded segments between splits planned to read them(vortex-layout): newPlannedSegments, one per file (keyed by its root layout, shared across scans via the footer cache, held weakly so it dies with the query). Each split registers the segments it announces and releases them when it finishes; a decoded segment is kept only while another live split registered it, and evicted when the last releases it. Global byte budgetVORTEX_SCAN_SHARED_SEGMENTS_MB(default 256), 16-way sharded map touched once per segment per split, prefetches skip already-decoded segments.VORTEX_SCAN_SHARED_SEGMENTS=0disables it. Previously each split re-fetched and re-deserialized segments other splits had already decoded (TPC-DS q09 decoded each segment ~21x across its 15 scans).Local results (4-core VM, DataFusion on Vortex, best of 3 interleaved rounds):
IO commit, after/before:
Shared segments, on/off (V2), and V2-on vs V1:
* driven by a few large DataFusion-bound queries whose rerun medians were within ~2%. Biggest wins: TPC-DS q09 69-77 ms → 57-62 ms (now level with V1), q92 25 → 20 ms. Peak RSS within +1-4% (ClickBench 2816 → 2927 MB); cache ops and lock waits < 0.3% of CPU in profiles.
Not run locally:
cargo clippy, unit tests (including the new ones), DuckDB benchmarks.🤖 Generated with Claude Code
https://claude.ai/code/session_016Ba4MkdFei46DRbqPuAQCW