Skip to content

do not merge(scan): V2 planning, local IO coalescing, and shared segment cache on top of #10339 - #10349

Closed
joseph-isaacs wants to merge 96 commits into
developfrom
ji/elegant-ritchie-wb6meo
Closed

joseph-isaacs wants to merge 96 commits into
developfrom
ji/elegant-ritchie-wb6meo

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Benchmark-only draft: this branch is #10339 plus three commits, opened against develop so the action/bench-sql label can run the SQL benchmarks (which already set VORTEX_SCAN_V2=1 on that branch). Only the three commits below are new relative to #10339.

Changes

  • perf(scan): avoid whole-struct expansion when pushing expressions through Pack (vortex-layout/src/plan/plans/pack.rs): ExpressionPackRule expands the struct root lazily (only the field accesses an expression uses), annotates free fields once, and checks identity expressions structurally. Trims V2 per-file planning on wide tables such as ClickBench hits.
  • perf(io): coalesce local reads across at most 64 KiB gaps (vortex-io): ObjectStoreReadAt over object_store's LocalFileSystem uses the local-file coalesce config instead of the object-storage one, and that config's gap shrinks from 1 MiB to 64 KiB (4 MiB cap unchanged). Remote stores are unchanged. With V2 announcing every segment of a split up front, the old gap let the coalescer chain gaps (ClickBench q20 read 430 MB to serve 277 MB of requested segments). Adds a unit test for the local-store detection.
  • perf(scan): share decoded segments between splits planned to read them (vortex-layout): new PlannedSegments, one per file (keyed by its root layout, shared across scans via the footer cache, held weakly so it dies with the query). Each split registers the segments it announces and releases them when it finishes; a decoded segment is kept only while another live split registered it, and evicted when the last releases it. Global byte budget VORTEX_SCAN_SHARED_SEGMENTS_MB (default 256), 16-way sharded map touched once per segment per split, prefetches skip already-decoded segments. VORTEX_SCAN_SHARED_SEGMENTS=0 disables it. Previously each split re-fetched and re-deserialized segments other splits had already decoded (TPC-DS q09 decoded each segment ~21x across its 15 scans).

Local results (4-core VM, DataFusion on Vortex, best of 3 interleaved rounds):

IO commit, after/before:

Suite V2 total V2 geomean V1 total V1 geomean
TPC-H SF1 1.006 0.990 0.975 0.975
TPC-DS SF1 0.991 0.972 0.984 0.962
ClickBench (20 of 100 shards) 0.975 0.952 0.973 0.965

Shared segments, on/off (V2), and V2-on vs V1:

Suite on/off total on/off geomean V2/V1 total V2/V1 geomean CPU off → on
TPC-H SF1 0.994 1.008 0.973 0.971 35.0s → 33.5s
TPC-DS SF1 0.990 0.974 0.997 0.971 381.5s → 378.0s
ClickBench (20 shards) 1.045* 1.000 0.997 0.959 161.5s → 162.8s

* driven by a few large DataFusion-bound queries whose rerun medians were within ~2%. Biggest wins: TPC-DS q09 69-77 ms → 57-62 ms (now level with V1), q92 25 → 20 ms. Peak RSS within +1-4% (ClickBench 2816 → 2927 MB); cache ops and lock waits < 0.3% of CPU in profiles.

Not run locally: cargo clippy, unit tests (including the new ones), DuckDB benchmarks.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Ba4MkdFei46DRbqPuAQCW

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge Pull requests that are not intended to merge stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants