Repository navigation
fix: Convert V2 scan batches to Arrow on the driver's tasks - #10861
Conversation
#10708 moved the decompress benchmark onto `ScanDriver`, which yields lazy `ArrayRef`s, and the benchmark then executed each one to Arrow on the single task polling the stream. Execution to Arrow is where the batches are decompressed, so decompression became serial, where the V2 `into_record_batch_stream` it replaced converted each batch on the chunk task that produced it. `ScanDriver::run_record_batches(builder, schema)` makes the driver the one place that maps batches: the caller names the Arrow schema it wants (or `None` to derive it from the scan's dtype) and the driver executes each batch to it on the producing task, in parallel across chunks. The decompress benchmark uses it. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBC5bWEJrRAdTA8Z5Bz
The CUDA decompress path in `gpu/vortex.rs` still uses it, which only builds with the `cuda` feature, so dropping it broke the all-features build. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBC5bWEJrRAdTA8Z5Bz
Polar Signals Profiling ResultsLatest Run
Previous Runs (8)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.014x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.985x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.137x ➖, 1↑ 5↓)
datafusion / parquet / ns (1.050x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed / ns (1.037x ➖, 2↑ 2↓)
duckdb / parquet / ns (1.064x ➖, 1↑ 3↓)
|
Merging this PR will degrade performance by 18.98%3 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.448x ✅, 25↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.586x ✅, 25↑ 0↓)
vortex / parquet / ns (0.999x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (0.993x ➖, 0↑ 0↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Resolve vortex-file's ScanDriver against #10861, which maps batches on the driver's tasks: keep develop's map_fn plumbing and reapply the per-root IO sessions, the inline-at-concurrency-one gate and the scan builder's concurrency. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GwhHqFVpQqPgQQkkK2y9kX
Summary
Fixes the decompress benchmark regression from #10708. That PR moved
compress-benchontoScanDriver::run, which yields lazyArrayRefs, and the benchmark then executed each batch to Arrow on the single task polling the stream. Executing to Arrow is where the batches are decompressed, so decompression became serial. The V2into_record_batch_streamit replaced converted each batch on the chunk task that produced it.This is a temporary fix, that will be improved on later..