Repository navigation
Reuse deserialized segments across partial V2 scans - #10849
joseph-isaacs wants to merge 1 commit into
Conversation
Merging this PR will improve performance by 15.16%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing |
Polar Signals Profiling ResultsLatest Run
Previous Runs (7)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.973x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.981x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.985x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.866x ➖, 2↑ 2↓)
datafusion / parquet / ns (0.950x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.920x ➖, 3↑ 0↓)
duckdb / parquet / ns (0.942x ➖, 1↑ 0↓)
|
Keep full encoded arrays in a per-file cache with four FIFO shards and a 16 MiB total weighted capacity. Partial scans can reuse deserialization and skip read requests on a hit; whole-segment scans bypass the cache. Decode, weigh, and release evicted arrays outside shard locks. Cover independent row selections, plan identity, disabled caching, failed reads, and eviction. Preserve dictionary-value reuse and integrate with the raw-segment retention controls: a zero byte budget disables both segment caches. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
a5e4a99 to
e4ce6e0
Compare
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
Summary
V2 scan chunks that overlap the same stored segment can repeatedly deserialize its encoded array and issue another segment request. Reuse the full encoded array across partial scans of a file, then apply each reader's own slice and mask.
Rebased onto fresh
developat1c2f6eb35304edfcc8a43a1bbdc3f667f267a4beon 29 October 2026. This diff adds only deserialized segment reuse; it introduces no dynamic-filter pushdown changes.Changes
ScanCache::with_retained_bytes(0)disables both segment caches; nonzero budgets keep the 16 MiB deserialized-array cache enabled.Performance evidence before rebase
These are local DataFusion measurements against
02d9b085a27c965cb6f1f9b652062e262dd11de0from #10799, with eight workers/partitions and V2 enabled on both sides. They have not been rerun against currentdevelop.The focused comparison used three alternating rounds of nine executions, discarding the first two executions in each process. Broader comparisons used three rounds of seven executions with the same warm-up exclusion. ClickBench Q2's profile showed 7.2% less worker CPU and segment-source compute CPU falling from 3.29 s to 0.94 s over 200 executions.
Known tradeoffs: isolated confirmation found FineWeb Q7 about 3.7% slower and TPC-H Q6 about 1.0% slower. One TPC-DS SF10 Q9 memory comparison increased peak process RSS by about 51 MiB. Cold TPC-DS results remained inconclusive after 11 pairs. These local measurements did not cover DuckDB or explicit segment-cache configurations.
Validation
vortex-layoutlibrary tests passed, including five new regression cases; pinned formatting and whitespace checks passed.git range-diffand resolved the overlap with develop's raw-segment retention controls. Preserved upstream's tests that disable segment retention and kept the decoded-cache regression cases.git diff --checkpasses and the worktree is clean.action/bench-sqlrun on rebased heade4ce6e00f2beb22b84b2e21592fd5a67974352ea. Builds and CI checks are pending; no local Rust build, tests, or clippy were run after this rebase. This remains a draft pending validation and review of the performance/memory tradeoffs.