Skip to content

Block fusion for ALP and DateTimeParts - #11542

Draft
joseph-isaacs wants to merge 13 commits into
claude/iterative-execute-vtable-benchmark-613qkp-executorfrom
claude/iterative-execute-vtable-benchmark-613qkp-ops
Draft

joseph-isaacs wants to merge 13 commits into
claude/iterative-execute-vtable-benchmark-613qkp-executorfrom
claude/iterative-execute-vtable-benchmark-613qkp-ops

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Part 2 of 3, stacked on #11541. It adds two more encodings as ordinary spine ops: ALP, and DateTimeParts over its days child. An encoded child streams into either of them block by block instead of being decoded into a full-length buffer first.

Changes

  • ALP. Its execute executes its patches first, then asks for encoded with require_spine!. Its block_op decodes each block of encoded integers into floats and patches it. If the patch values contain nulls, it declines and runs level-wise, which also patches validity.
  • DateTimeParts.
    • Its execute decodes seconds and subseconds first, then asks for days with require_spine!.
    • Its block_op for days combines each block of days with the same rows of the decoded time parts, using the same combine_parts kernel as decode_to_temporal.
    • Days encoded as FoR(BitPacked) is the leaf, so days is never materialized: at the root, or under any parent that extends the chain.
    • Time parts that aren't i32, and the Days unit, have no op, so they decode level-wise.
  • vortex-array: extension types stored as primitives can now fuse. The driven result is wrapped in the extension dtype, time zone included.
  • An earlier version of this PR added a separate combine API (block_combine, ZipSource, try_combine). It's gone: with a leaf being anything that has a block source, DateTimeParts needs nothing beyond block_op.

Tests

Every test compares fused output with fusion off and asserts the full trace.

  • ALP: ALP(FoR(BitPacked)) on f64 and f32, plain, patched, and nullable with patches. Slices cover mid-chunk, ending just past a patch, and f32. ALP over a canonical child runs its own kernel.
  • DateTimeParts:
    • days as FoR(BitPacked), plain, nullable and with a time zone;
    • three slices;
    • constant time parts, which execute first, as level-wise decoding does;
    • Dict seconds, which execute first, and then days still fuses;
    • all-canonical children, which run level-wise;
    • i64 seconds, where no op means no source is built.

Benchmark

Measured locally (15 rounds × 100 samples, earlier revision). Speedup is level-wise ÷ fused.

tree 64Ki rows 4Mi rows
ALP(FoR(BP)) f64 1.27x 2.03x
ALP(FoR(BP)) f64, patched every 997 rows 1.32x 2.08x
DateTimeParts, FoR(BP) days 1.24x 2.17x

The DateTimeParts numbers were measured with the earlier combine design and not re-measured since.

Checks

  • cargo test -p vortex-array -p vortex-fastlanes -p vortex-alp -p vortex-datetime-parts --lib
  • cargo clippy -p vortex-array -p vortex-fastlanes -p vortex-alp -p vortex-datetime-parts --all-targets --all-features -- -D warnings
  • cargo +nightly-2026-11-13 fmt
  • Not run: typos, docs build, workspace-wide checks, benchmarks on the current head.

API Changes

  • Behaviour change: ALP executes its patches before its encoded child, and DateTimeParts executes its time parts before days. Results are unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U

Builds on the executor's block fusion with ops for three more encodings:

- ALP decodes each block of encoded integers into floats and patches it.
  ALP's execute now executes its patches before requesting its encoded
  child as its spine.
- Cast: a cast that cannot fail (a lossless primitive conversion that
  does not drop nullability) asks for its input as its spine, until the
  input is columnar, and converts blocks value by value. Other casts
  execute as before.
- ZigZag requests its encoded child as its spine instead of executing it
  recursively, and decodes blocks of unsigned integers.

The cast op uses the child dtype block_op receives to check the
conversion is lossless without reading the taken slot.

Adds ALP, cast and ZigZag cases to the block_fusion benchmark and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs and others added 7 commits November 17, 2026 16:41
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

# Conflicts:
#	encodings/fastlanes/benches/block_fusion.rs
ALP decodes and patches each block of its encoded spine as a block op, so
ALP over FoR over BitPacked runs as one fused pass.

DateTimeParts combines blocks of its days, seconds and subseconds children
with a new BlockCombine, which the executor fuses where the array stands:
canonical children are lent from their buffers and block-source children
(such as FoR over BitPacked days) fill per block, so no child is
materialized. Extension dtypes stored as primitives can now be fused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- Combine a DateTimeParts array only when a child is a block source:
  combining canonical children saves no buffer and costs a copy.
- DateTimeParts executes its time parts before days, so a days child that
  is a block source is combined rather than decoded on its own.
- Check every child is ready before building any source, and decline a
  child of the wrong length instead of failing.
- Combine constant time parts only when they are i32, as level-wise
  decoding reads them.
- try_combine asks for the combine before the kernel lookups, and skips
  fused arrays.
- ZipSource keeps its per-block pointers on the stack, and the fused
  extension array is built fallibly.
- ALP declines fusion when its patch values hold nulls.
- Tests: assert the full traces, including that no child executes on its
  own, over time zones, slices, f32, sliced patches, constant parts, and
  time parts that must execute first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A block program with a combine and no op copied each combined block from
scratch into the output through an identity op. The combine now writes
each block into the output directly, which saves a pass per block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs joseph-isaacs changed the title Fuse ALP decodes, lossless casts and ZigZag decodes as block ops Block fusion for ALP and DateTimeParts Nov 17, 2026
Combine into a FusedChain, which extends with the parents above and drives, instead of into a Fused array.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
joseph-isaacs and others added 4 commits November 19, 2026 10:52
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
With a leaf being anything that has a block source, DateTimeParts needs
no combine machinery of its own. Its execute decodes seconds and
subseconds first and asks for days as its spine; its block_op for days
combines each block with the same rows of the decoded time parts. An
encoded days child is the leaf, so it is never materialized, at the root
or under any parent that extends the chain.

This removes VTable::block_combine, BlockCombine, BlockCombineSpec,
ZipSource, the zip program input, FusedChain::combine and try_combine.
What remains in vortex-array is fusing extension types stored as
primitives. Constant time parts now execute into buffers first, as
level-wise decoding already did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant