Repository navigation
feat(layout): run plans as push-based pipelines over a scan's splits - #10420
joseph-isaacs wants to merge 1 commit into
Conversation
Adds `plan::pipeline`, the executor the design document describes, beside the node-based `plan::exec`: - Operator and Source traits with one compute method returning a Step, sources blocking only on IO, an empty inlet, or a full outlet. - Single-writer, single-reader ports in a scan-owned arena, with the capacity the reading source declares. - A driver that pushes each batch through a pipeline's stages and a scheduler that wakes a pipeline only when its blocking condition clears. - A Scan over a list of splits: a Query runs as stages (zone pruning, each conjunct under the rows the earlier ones kept, the projection), every mask is known when its stage is compiled, and chunks a mask selects nothing of are never built or read. - Shared decoding: one pass over the stage plans records the rows each segment is read for; the first reader of a segment read more than once builds one decoding pipeline that fans out into a single-use port per reader, tagged by split, and a split's unclaimed ports are dropped when it finishes. The query benchmark and profiling example gain a Pipeline algorithm, and the example can alternate two exec algorithms (ALGO2). Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NWYmfHTFtaEdhdyd3u5hv3
Merging this PR will degrade performance by 36.21%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | bench_compare_varbinview[(50000, 2)] |
144.8 µs | 400.6 µs | -63.85% |
| ❌ | Simulation | bench_compare_varbinview[(50000, 4)] |
144.8 µs | 400.5 µs | -63.84% |
| ❌ | Simulation | bench_compare_varbinview[(50000, 8)] |
145.7 µs | 400.6 µs | -63.63% |
| ❌ | Simulation | case_when_fragmented[400] |
183.9 µs | 503.9 µs | -63.51% |
| ❌ | Simulation | bench_compare_varbinview[(50000, 32)] |
146.5 µs | 401.6 µs | -63.51% |
| ❌ | Simulation | bench_compare_varbinview[(50000, 128)] |
148.4 µs | 404.7 µs | -63.32% |
| ❌ | Simulation | bench_compare_primitive[(50000, 4)] |
163.2 µs | 418.8 µs | -61.03% |
| ❌ | Simulation | bench_compare_primitive[(50000, 2)] |
163.1 µs | 418.5 µs | -61.03% |
| ❌ | Simulation | bench_compare_primitive[(50000, 8)] |
164 µs | 418.9 µs | -60.84% |
| ❌ | Simulation | bench_compare_primitive[(50000, 32)] |
164.8 µs | 419.8 µs | -60.74% |
| ❌ | Simulation | bench_compare_primitive[(50000, 128)] |
165.9 µs | 422 µs | -60.69% |
| ❌ | Simulation | bench_compare_varbin[(50000, 2)] |
168.4 µs | 424.2 µs | -60.31% |
| ❌ | Simulation | bench_compare_varbin[(50000, 4)] |
168.6 µs | 424.7 µs | -60.3% |
| ❌ | Simulation | bench_compare_varbin[(50000, 8)] |
169.6 µs | 424.6 µs | -60.06% |
| ❌ | Simulation | bench_compare_varbin[(50000, 32)] |
170.3 µs | 425.5 µs | -59.97% |
| ❌ | Simulation | bench_compare_varbin[(50000, 128)] |
172.4 µs | 428.7 µs | -59.79% |
| ❌ | Simulation | bench_compare_primitive[(50000, 2048)] |
216.2 µs | 502 µs | -56.94% |
| ❌ | Simulation | bench_compare_varbinview[(50000, 512)] |
198 µs | 455.1 µs | -56.5% |
| ❌ | Simulation | bench_compare_varbinview[(50000, 2048)] |
220.2 µs | 502.3 µs | -56.16% |
| ❌ | Simulation | bench_compare_primitive[(50000, 512)] |
209.6 µs | 472.3 µs | -55.61% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/exec-7-pipeline (c8ba445) with develop (3d12093)2
Footnotes
-
775 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
ji/exec-6-execute-stream(6eb349c) during the generation of this report, sodevelop(3d12093) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Summary
Part of the exec plan stack extracted from #10349. Stacked on #10419.
This PR implements the pipeline executor described in
vortex-layout/docs/pipeline-exec-design.mdasplan::pipeline. It sits beside the node-basedplan::exec, so the benchmarks can compare the two.Changes
Operatorhas onecomputemethod that returns aStep:More,Last,Consumed,BlockedorFinished.Sourceextends it with inlets, a declared inlet capacity, and segment requests. Only a source blocks, and only on IO, an empty inlet, or a full outlet.peek_mutandtakeon its inlet. The reading source sets the capacity.Queryruns per split as stages: zone pruning, then each conjunct under the rows the earlier ones kept, then the projection. Every mask is known when its stage is compiled, so a chunk the mask selects nothing of is never built or read.Pipelinealgorithm. The example can alternate two exec algorithms withALGO2.Tests
The node executor's cases are ported, about 90 tests, and run with FIFO and LIFO delivery. New tests check three things:
🤖 Generated with Claude Code
https://claude.ai/code/session_01NWYmfHTFtaEdhdyd3u5hv3
Generated by Claude Code