Repository navigation
feat: add fused block executor vtable method - #11541
joseph-isaacs wants to merge 16 commits into
Conversation
The iterative executor descends an encoding tree with ExecuteSlot frames, so when it reaches a leaf, the frames on its stack are the parents waiting for that leaf to be canonical. Level-wise execution writes a full-length buffer at every level of that chain. The executor now fuses it instead: the leaf emits blocks of at most 1024 rows (BlockSource), each waiting parent transforms a block (BlockOp) in L1 scratch, and only the outermost parent writes a full-length buffer. An encoding opts in explicitly. It asks for the last child its execute waits for with the new ExecutionStep::ExecuteSpine (execute_spine, require_spine!), promising that every other child it needs is executed and that once this child is canonical it only transforms the child's values row by row, as its VTable::block_op does. Leaves implement VTable::block_source. The executor only fuses through spine frames. execute_until tries fusion before executing an encoded leaf, after its execute_parent probes so no parent kernel is skipped, and when a canonical child has two or more spine parents waiting above it. A leaf with no block source but an already canonical child uses that child as the source with its own op on top. Anything else executes level-wise as before. Validity is not streamed; it is taken from the outermost parent. Implemented for Primitive and BitPacked (sources) and FoR: a source over BitPacked when their chunks line up, running the fused unfor_pack per chunk, and a reference-adding op over any other child. FoR requests its encoded child as its spine. Ops map blocks with the same lane kernels level-wise execution uses. Disable with ExecutionCtx::with_block_fusion(false) or VORTEX_BLOCK_FUSION=0. Each fused chain logs a vortex_array::block debug event. Adds encodings/fastlanes/benches/block_fusion.rs, comparing fused and level-wise execution on the same arrays, and documents block fusion in the execution internals guide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Drop FoR's own block source, which duplicated the BitPacked unpack with FoR references threaded through it. A leaf without a block source now fuses its own op over any child that has one, not only a canonical child, so FoR over BitPacked fuses as BitPacked's chunk-by-chunk source with FoR's add-reference op on top, and BitPacked's source no longer knows about FoR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
FoR over BitPacked with one global reference is a block source again, unpacking each chunk with FastLanes' fused unpack-and-add, as FoR's own execute does, instead of unpacking and then adding the reference in a second pass over each block. The BitPacked block source is now generic over the existing UnpackStrategy the level-wise unpack uses: BitPacked uses BitPackingStrategy and FoR a new FoRStrategy, so the chunk loop is not duplicated. FoR with per-chunk references still fuses as BitPacked's source with FoR's op. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will degrade performance by 0.49%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.980x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.009x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Its fused and level-wise pairs were for measuring the change during development; fusion can be toggled with VORTEX_BLOCK_FUSION=0 on any existing benchmark. The existing decompress[alp_for_bp_f64] encoding-tree benchmark covers the tree that fuses in practice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.012x ➖, 0↑ 3↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.987x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.928x ➖, 4↑ 3↓)
datafusion / parquet / ns (0.963x ➖, 2↑ 1↓)
duckdb / vortex-file-compressed / ns (1.128x ➖, 1↑ 3↓)
duckdb / parquet / ns (0.916x ➖, 2↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.992x ➖, 1↑ 0↓)
No file size changes detected. |
BlockOp::process now takes a Block that is either Copy (an input and separate output memory) or InPlace (one buffer the op overwrites), and returns a Written proof. Block::map writes every value with the lane kernels for either shape, so ops that only map, like FoR's, need no change for in-place blocks, and an op can only skip values with unsafe code, closing the hole where a safe op left output uninitialized. The executor overwrites scratch in place wherever an op keeps its width, so a chain needs one scratch block, and only the last op copies into the output. A canonical leaf it holds the only reference to is decoded in its own buffer when every op keeps the width, as level-wise execution does, instead of into a second full-length buffer. execute_until releases its input once optimized so that leaf can be uniquely owned. Planning finds the chain's input before building any op, so a leaf that cannot fuse costs no op construction. Primitive no longer needs a block source: the executor reads canonical arrays itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The trace harness records ExecuteSpine as its own step, each fused chain (the outermost array, whether its input is a source or a canonical leaf, and how many frames and ops it consumed), and, for the first block, how each op ran: in place or copying, and which memory it read and wrote (the source's own memory, a scratch block, the output, or the leaf's buffer). Trace events now render the arrays they describe when recorded instead of holding clones of them, so tracing does not keep arrays alive and change what it observes: a leaf the executor owns uniquely is still decoded in place while traced. New tests assert the schedules of FoR stacks over BitPacked, FoR over a shared and an owned canonical leaf, and FoR with misaligned per-chunk references. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Replace planning from the leaf with fusion that follows the executor's trampoline. The loop descends with ExecuteSpine frames as before. On the way back up, a child that can emit blocks (a block source, or a canonical primitive array) starts a block program when the spine parent waiting for it has a block op, and the parent becomes a Fused array carrying the program. Each spine parent above extends it with its op. When the next parent cannot extend it, or at the root, executing the Fused array drives the whole program in one pass, and execution carries on. Fused is an internal encoding with no slots that never leaves execute_until. Each fused parent takes its validity and statistics from its own vtables with the child back in its slot, so validity is derived per level, as level-wise execution does. The leaf-time planner, its canonical-leaf special case and its minimum-length rules go away. BlockSource::fill now takes the rows to emit, and block_offset reports the source's grid, so the executor chooses the blocks: on the source's grid, a BitPacked range lies in one chunk and is lent from it; a range across chunks is assembled. Block is #[non_exhaustive]. Traces record fuse start, fuse extend and drive. New executor tests with test encodings assert the build and drive order, copying and in-place schedules, width changes, ops without in-place support, a non-spine parent driving the chain below it, the block grid, owned and shared canonical leaves, disabled fusion and empty arrays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- In-place ops wrote through pointers derived from a shared borrow, which is undefined behaviour. BlockSource::fill now returns a SourceBlock: values written into the executor's scratch are addressed through the executor's own mutable borrow and may be overwritten in place, while values lent from the source's memory are only read. The read-only BlockBuf::view is gone. - A fused array is never done for the target or a frame's predicate, so it is always driven before execute_until returns, whatever the target matches. - A parent with execute-parent kernels of its own, or one the next parent up has a kernel for, is not fused, so those kernels still probe it with its child executed. One try_fuse helper serves both call sites. - Fused results materialize validity at drive, as level-wise execution does. - fuse_into_parent checks the child is primitive and has its parent's length before asking for the op, and finds the program's input before building the op. - The second scratch block is allocated only when an op copies between scratch blocks. - BlockPatches keeps indices in their stored type, checks indices and values match in length and values are all valid. - FoR with a zero global reference does not fuse, keeping its zero-copy level-wise path. - check_written reports lengths and types with vortex_ensure_eq. - Shared-leaf tests check the leaf's values are untouched; new tests cover a target matching a fused array and a parent with kernels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
execute_encoding_unchecked called the encoding through a shared borrow of the array's data while also moving in the ArrayRef that owns it. When that ArrayRef was the last reference, the step dropped the array while the borrow was still a live function argument, which Miri reports as undefined behaviour. It was latent while execute_until held its input for the whole loop, and reachable now that it releases it. DynArrayData::execute_fn returns the step as a function taking the array by value, so the borrow ends before the step runs. The checked execute_encoding applies its postconditions around it, and DynArrayData::execute and execute_unchecked go away. The block tests pass under Miri. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A block source claims its scratch holds initialized values when it returns SourceBlock::Scratch, which safe code could do without writing them. The variant is now non-exhaustive and built with the unsafe SourceBlock::scratch, as op outputs already need Written::assume_init. BlockPatches::try_new returns None for null patch values instead of failing, so such an array executes level-wise, which patches validity too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…ource The block source unpacked chunks and checked the packed buffer's length with its own copies of what UnpackedChunks does. Both now unpack through one safe unpack_chunk_into, which bounds the chunk and the scratch it writes, and the source checks its buffer with validate_packed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A Fused array never waited: as soon as it existed, the executor extended it with the next spine parent or drove it. It was an array only so it could travel through the loop, which cost an encoding, a mutex, and a special case in the done check so it never escaped execute_until. try_fuse now starts a FusedChain, extends it with each spine parent that can, popping their frames, and drives it right away. The parent's validity, which the Fused array got by standing in for the taken slot, is taken when a spine frame is pushed, while the slot is in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Every check that does not need the slot executed now runs when the parent asks for its spine: both arrays primitive, non-empty and of one length, no builder, no execute-parent kernels on the parent, and no kernel the next parent up has for it. A slot that fails runs as a plain ExecuteSlot, so require_spine! needs no check of its own, and the way up spends nothing on it: no validity is taken, no source is built, and finding a fusable frame is a field read. Block ops are still built on the way up. Traces print such a slot as ExecuteSlot, so tests assert the decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A program's input came from three paths: a canonical primitive array, an array's block source, or an array whose own op transformed a canonical child. A leaf is now one check, block_source: Primitive implements it by lending its buffer, and decoding in place is the optional BlockSource::take_buffer, which a source whose buffer nothing else references answers. The third path is gone: the innermost op over a primitive array runs its own execute, and its result is the leaf of the ops above, decoded in place since the executor owns it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- A chain takes the validity of its leaf on the way up, instead of the parent's validity at push time. A spine parent's validity is its spine child's (now part of the ExecuteSpine and block_op contracts), and taking it from the executed leaf no longer executes a lazy validity twice: FoR over a Dict with nullable values decoded the Dict's codes twice when fused. Frames keep only a fuse flag. - BlockProgram holds its source directly: ProgramInput had one variant. Traces drop the input= field it named. - BlockPatches keeps its patches as one list of rows and values, instead of indices in their stored type and a four-way match. - Fusion is on by default and switched with with_block_fusion only: the VORTEX_BLOCK_FUSION variable and the unused getter are gone. - Docs: a leaf is anything with a block source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
On the way up the executor built the leaf's source first, and only then asked the parent for its op, so a parent without one (FoR with a zero reference, ALP with null patch values) cost a source, patches included, on every pass. Now it asks for the op first. A parent without one turns its frame off, so later passes are a field read; an op built while the child is not yet a leaf waits in the frame until the child executes, so it is never built twice. FusedChain is now start, push and drive: the executor finds each parent's op, and the chain only records it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Part 1 of 3. This PR adds block fusion to the iterative executor, plus the BitPacked and FoR implementations. #11542 adds ALP and DateTimeParts on top, and #11586 (draft) drives chains straight into builders. It revisits the 1024-row streaming idea from #9614, built into the iterative executor instead of a separate recursive vtable path.
Level-wise execution writes a full-length buffer at every level of an encoding chain such as FoR(FoR(BitPacked)). With this PR, the leaf emits blocks of at most 1024 rows and each parent transforms a block while it is in L1. Only the outermost parent writes a full-length buffer.
Changes
How an encoding takes part
require_spine!(ExecutionStep::ExecuteSpine). An encoding uses it for the last child itsexecutewaits for. It promises that once that child is canonical, the encoding only transforms the child's values row by row, and that its validity is the child's. When the slot can't fuse, it behaves exactly likerequire_child!, so encodings never check fusability themselves.VTable::block_op(slot, child_dtype): the row-by-row transform, as aBlockOpover one block.Block::mapruns it either in place or copying.VTable::block_source: a leaf is anything whoseblock_sourcereturnsSome, and that is the only check. Primitive lends its buffer, BitPacked unpacks one chunk at a time, and FoR over BitPacked with one global reference uses FastLanes' fused unpack-and-add.What the executor does
Going down: check. When a frame is pushed for
ExecuteSpine, the executor checks everything that doesn't need the child executed:A slot that fails runs as a plain
ExecuteSlot, and the trace printsExecuteSlot.Going up: build.
FusedChainstarts from its source. The op of each fusable frame above is then pushed onto the chain.Drive. As soon as the next frame can't extend the chain, the program runs block by block:
A chain is never left waiting, so it is a value in the loop, not an array.
Fusion runs after an edge's
execute_parentprobes, so no kernel is skipped. Disable it withExecutionCtx::with_block_fusion(false).Docs:
docs/developer-guide/internals/execution.mdgains a Block Fusion section.Tests
Every test compares fused output with fusion off, and asserts the trace in full:
ExecuteSpine/ExecuteSlot,fuse start,fuse extend,drive, and how each op ran over the first block. Cases:Benchmark
Measured locally with a fused vs level-wise benchmark that is not committed (40 rounds × 100 samples). Speedup is level-wise ÷ fused. These numbers come from an earlier revision of this PR: the structure has changed since (validity from the leaf, op before source, primitive as an ordinary leaf) and I haven't re-measured. FoR⁴(Primitive) in particular fuses one level fewer now.
Checks
cargo test -p vortex-array -p vortex-fastlanes --libcargo clippy -p vortex-array -p vortex-fastlanes --all-targets --all-features -- -D warningscargo +nightly-2026-11-13 fmttypos, docs build, workspace-wide checks, benchmarks on the current head.API Changes
vortex_array::blockmodule:BlockSource,BlockOp,Block,Written,SourceBlock,BlockPatches.ExecutionStep::ExecuteSpine, which breaks exhaustive matches onExecutionStep.ExecutionResult::execute_spineandrequire_spine!.VTable::block_sourceandVTable::block_op, both with defaults.ExecutionCtx::with_block_fusion.🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U