Skip to content

Drive fused block chains straight into a builder - #11586

Draft
joseph-isaacs wants to merge 3 commits into
claude/iterative-execute-vtable-benchmark-613qkp-opsfrom
claude/iterative-execute-vtable-benchmark-613qkp-sink
Draft

joseph-isaacs wants to merge 3 commits into
claude/iterative-execute-vtable-benchmark-613qkp-opsfrom
claude/iterative-execute-vtable-benchmark-613qkp-sink

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Part 3 of 3, stacked on #11542 (which is stacked on #11541). This draft checks that block fusion works with a builder as the sink: a fused chain that stands for a whole array can write straight into the builder that array is being appended to, for example each chunk of a Chunked array.

Without it, a chunk that has no append_to_builder of its own is decoded into its own buffer, which is then copied into the builder.

Changes

  • Executor: when a chain is extended all the way to the root, the loop returns it undriven, and each caller decides where it goes:

    • execute_until drives it into a buffer, as before;
    • ArrayRef::execute_and_append, used by the default append_to_builder, drives it straight into the builder. If the builder isn't a primitive builder of the chain's dtype, it drives into a buffer and appends that.

    The executor itself never handles builders.

  • FusedChain::drive_into: reserves room first, writes the values into the builder's uninitialized tail through BlockProgram::write_into, then appends the validity.

  • Traces mark these drives with into=builder.

  • Encodings with their own append_to_builder (BitPacked, Dict, Primitive, …) are unaffected.

Tests

  • chunks_drive_into_the_builder (vortex-array), with the full trace asserted. Four chunks:
    • a two-op chain, driven into the builder;
    • a one-op chain, driven into the builder;
    • a chain under a non-spine parent, which stops below its chunk's root and gets its own buffer;
    • a plain chunk, appended as before.
  • chunks_without_fusion_append_level_wise.
  • chunks_drive_into_the_builder (FastLanes): Chunked of FoR stacks with nulls, patches and a sliced chunk. Every drive goes into the builder, and the result matches level-wise.
  • drives_into_a_builder_without_capacity: a regression test for the review finding that appending a fused stack into an empty PrimitiveBuilder panicked.

Checks

  • cargo test -p vortex-array -p vortex-fastlanes -p vortex-alp -p vortex-datetime-parts --lib
  • cargo clippy -p vortex-array -p vortex-fastlanes -p vortex-alp -p vortex-datetime-parts --all-targets --all-features -- -D warnings
  • cargo +nightly-2026-11-13 fmt
  • Not run: benchmarks, Miri, typos, docs build, workspace-wide checks.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U

joseph-isaacs and others added 3 commits November 18, 2026 10:27
…ded to

A chunked array appends each chunk to one builder. A chunk without its own
append_to_builder executed into a buffer of its own, which was then copied
into the builder. The default append_to_builder now runs the executor's
loop with the builder as a sink: a fused chain that stands for the chunk is
driven into the builder's uninitialized tail, and anything else is appended
as before.

BlockProgram::write_into runs a program into any slice of slots, and
FusedChain::drive_into uses it for a primitive builder of the chain's
dtype. Traces mark such drives with into=builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Rebuilt the builder sink on the current executor:

- The loop returns a fused chain that reaches the root undriven, instead
  of taking a sink parameter through every fusion function. execute_until
  drives it into a buffer; execute_and_append, the default
  append_to_builder, drives it into the builder, or into a buffer it then
  appends when the builder is not a primitive builder of its dtype. The
  executor knows nothing of builders.
- drive_into reserves room for its rows first. Appending a fused FoR stack
  into a builder with no spare capacity panicked.
- BlockProgram::write_into runs a program into any slice of slots.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant