Skip to content

feat(layout): run plans as push-based pipelines over a scan's splits - #10420

Closed
joseph-isaacs wants to merge 1 commit into
ji/exec-6-execute-streamfrom
ji/exec-7-pipeline
Closed

joseph-isaacs wants to merge 1 commit into
ji/exec-6-execute-streamfrom
ji/exec-7-pipeline

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

Part of the exec plan stack extracted from #10349. Stacked on #10419.

This PR implements the pipeline executor described in vortex-layout/docs/pipeline-exec-design.md as plan::pipeline. It sits beside the node-based plan::exec, so the benchmarks can compare the two.

Changes

  • Traits. Operator has one compute method that returns a Step: More, Last, Consumed, Blocked or Finished. Source extends it with inlets, a declared inlet capacity, and segment requests. Only a source blocks, and only on IO, an empty inlet, or a full outlet.
  • Ports. Each port has one writer and one reader and lives in an arena the scan owns. A reader uses peek_mut and take on its inlet. The reading source sets the capacity.
  • Driver and scheduler. The driver pushes each batch through a pipeline's stages into its outlets. The scheduler runs the most recently woken pipeline first and wakes a blocked pipeline only when its condition clears.
  • Scan over splits. A Query runs per split as stages: zone pruning, then each conjunct under the rows the earlier ones kept, then the projection. Every mask is known when its stage is compiled, so a chunk the mask selects nothing of is never built or read.
  • Shared decoding. One pass over the stage plans records the rows each segment is read for. The first reader of a segment read more than once builds one decoding pipeline. That pipeline fans out into a single-use port per reader, tagged by split. A split's unclaimed ports are dropped when it finishes. This covers columns read by both a conjunct and the projection, segments wider than a split, zone tables and dictionary values.
  • Benchmarks. The query benchmark and the profiling example gain a Pipeline algorithm. The example can alternate two exec algorithms with ALGO2.

Tests

The node executor's cases are ported, about 90 tests, and run with FIFO and LIFO delivery. New tests check three things:

  • several splits in one scan read each segment once;
  • a query reads a column it both filters and projects only once;
  • a finished scan holds no port and no unclaimed share.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NWYmfHTFtaEdhdyd3u5hv3


Generated by Claude Code

Adds `plan::pipeline`, the executor the design document describes,
beside the node-based `plan::exec`:

- Operator and Source traits with one compute method returning a Step,
  sources blocking only on IO, an empty inlet, or a full outlet.
- Single-writer, single-reader ports in a scan-owned arena, with the
  capacity the reading source declares.
- A driver that pushes each batch through a pipeline's stages and a
  scheduler that wakes a pipeline only when its blocking condition
  clears.
- A Scan over a list of splits: a Query runs as stages (zone pruning,
  each conjunct under the rows the earlier ones kept, the projection),
  every mask is known when its stage is compiled, and chunks a mask
  selects nothing of are never built or read.
- Shared decoding: one pass over the stage plans records the rows each
  segment is read for; the first reader of a segment read more than
  once builds one decoding pipeline that fans out into a single-use
  port per reader, tagged by split, and a split's unclaimed ports are
  dropped when it finishes.

The query benchmark and profiling example gain a Pipeline algorithm,
and the example can alternate two exec algorithms (ALGO2).

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NWYmfHTFtaEdhdyd3u5hv3
@codspeed

codspeed Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 36.21%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 10 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 6 improved benchmarks
❌ 68 regressed benchmarks
✅ 1693 untouched benchmarks
🆕 22 new benchmarks
⏩ 775 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation bench_compare_varbinview[(50000, 2)] 144.8 µs 400.6 µs -63.85%
❌ Simulation bench_compare_varbinview[(50000, 4)] 144.8 µs 400.5 µs -63.84%
❌ Simulation bench_compare_varbinview[(50000, 8)] 145.7 µs 400.6 µs -63.63%
❌ Simulation case_when_fragmented[400] 183.9 µs 503.9 µs -63.51%
❌ Simulation bench_compare_varbinview[(50000, 32)] 146.5 µs 401.6 µs -63.51%
❌ Simulation bench_compare_varbinview[(50000, 128)] 148.4 µs 404.7 µs -63.32%
❌ Simulation bench_compare_primitive[(50000, 4)] 163.2 µs 418.8 µs -61.03%
❌ Simulation bench_compare_primitive[(50000, 2)] 163.1 µs 418.5 µs -61.03%
❌ Simulation bench_compare_primitive[(50000, 8)] 164 µs 418.9 µs -60.84%
❌ Simulation bench_compare_primitive[(50000, 32)] 164.8 µs 419.8 µs -60.74%
❌ Simulation bench_compare_primitive[(50000, 128)] 165.9 µs 422 µs -60.69%
❌ Simulation bench_compare_varbin[(50000, 2)] 168.4 µs 424.2 µs -60.31%
❌ Simulation bench_compare_varbin[(50000, 4)] 168.6 µs 424.7 µs -60.3%
❌ Simulation bench_compare_varbin[(50000, 8)] 169.6 µs 424.6 µs -60.06%
❌ Simulation bench_compare_varbin[(50000, 32)] 170.3 µs 425.5 µs -59.97%
❌ Simulation bench_compare_varbin[(50000, 128)] 172.4 µs 428.7 µs -59.79%
❌ Simulation bench_compare_primitive[(50000, 2048)] 216.2 µs 502 µs -56.94%
❌ Simulation bench_compare_varbinview[(50000, 512)] 198 µs 455.1 µs -56.5%
❌ Simulation bench_compare_varbinview[(50000, 2048)] 220.2 µs 502.3 µs -56.16%
❌ Simulation bench_compare_primitive[(50000, 512)] 209.6 µs 472.3 µs -55.61%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/exec-7-pipeline (c8ba445) with develop (3d12093)2

Open in CodSpeed

Footnotes

  1. 775 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on ji/exec-6-execute-stream (6eb349c) during the generation of this report, so develop (3d12093) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@robert3005
robert3005 deleted the ji/exec-7-pipeline branch November 10, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants