Repository navigation
Block fusion for ALP and DateTimeParts - #11542
Draft
joseph-isaacs wants to merge 13 commits into
Draft
joseph-isaacs wants to merge 13 commits into
joseph-isaacs wants to merge 13 commits into
Conversation
Builds on the executor's block fusion with ops for three more encodings: - ALP decodes each block of encoded integers into floats and patches it. ALP's execute now executes its patches before requesting its encoded child as its spine. - Cast: a cast that cannot fail (a lossless primitive conversion that does not drop nullability) asks for its input as its spine, until the input is columnar, and converts blocks value by value. Other casts execute as before. - ZigZag requests its encoded child as its spine instead of executing it recursively, and decodes blocks of unsigned integers. The cast op uses the child dtype block_op receives to check the conversion is lossless without reading the taken slot. Adds ALP, cast and ZigZag cases to the block_fusion benchmark and tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> # Conflicts: # encodings/fastlanes/benches/block_fusion.rs
ALP decodes and patches each block of its encoded spine as a block op, so ALP over FoR over BitPacked runs as one fused pass. DateTimeParts combines blocks of its days, seconds and subseconds children with a new BlockCombine, which the executor fuses where the array stands: canonical children are lent from their buffers and block-source children (such as FoR over BitPacked days) fill per block, so no child is materialized. Extension dtypes stored as primitives can now be fused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- Combine a DateTimeParts array only when a child is a block source: combining canonical children saves no buffer and costs a copy. - DateTimeParts executes its time parts before days, so a days child that is a block source is combined rather than decoded on its own. - Check every child is ready before building any source, and decline a child of the wrong length instead of failing. - Combine constant time parts only when they are i32, as level-wise decoding reads them. - try_combine asks for the combine before the kernel lookups, and skips fused arrays. - ZipSource keeps its per-block pointers on the stack, and the fused extension array is built fallibly. - ALP declines fusion when its patch values hold nulls. - Tests: assert the full traces, including that no child executes on its own, over time zones, slices, f32, sliced patches, constant parts, and time parts that must execute first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A block program with a combine and no op copied each combined block from scratch into the output through an identity op. The combine now writes each block into the output directly, which saves a pass per block. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Combine into a FusedChain, which extends with the parents above and drives, instead of into a Fused array. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
With a leaf being anything that has a block source, DateTimeParts needs no combine machinery of its own. Its execute decodes seconds and subseconds first and asks for days as its spine; its block_op for days combines each block with the same rows of the decoded time parts. An encoded days child is the leaf, so it is never materialized, at the root or under any parent that extends the chain. This removes VTable::block_combine, BlockCombine, BlockCombineSpec, ZipSource, the zip program input, FusedChain::combine and try_combine. What remains in vortex-array is fusing extension types stored as primitives. Constant time parts now execute into buffers first, as level-wise decoding already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 2 of 3, stacked on #11541. It adds two more encodings as ordinary spine ops: ALP, and DateTimeParts over its days child. An encoded child streams into either of them block by block instead of being decoded into a full-length buffer first.
Changes
executeexecutes its patches first, then asks forencodedwithrequire_spine!. Itsblock_opdecodes each block of encoded integers into floats and patches it. If the patch values contain nulls, it declines and runs level-wise, which also patches validity.executedecodes seconds and subseconds first, then asks for days withrequire_spine!.block_opfor days combines each block of days with the same rows of the decoded time parts, using the samecombine_partskernel asdecode_to_temporal.i32, and theDaysunit, have no op, so they decode level-wise.block_combine,ZipSource,try_combine). It's gone: with a leaf being anything that has a block source, DateTimeParts needs nothing beyondblock_op.Tests
Every test compares fused output with fusion off and asserts the full trace.
i64seconds, where no op means no source is built.Benchmark
Measured locally (15 rounds × 100 samples, earlier revision). Speedup is level-wise ÷ fused.
The DateTimeParts numbers were measured with the earlier combine design and not re-measured since.
Checks
cargo test -p vortex-array -p vortex-fastlanes -p vortex-alp -p vortex-datetime-parts --libcargo clippy -p vortex-array -p vortex-fastlanes -p vortex-alp -p vortex-datetime-parts --all-targets --all-features -- -D warningscargo +nightly-2026-11-13 fmttypos, docs build, workspace-wide checks, benchmarks on the current head.API Changes
🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U