Repository navigation
Drive fused block chains straight into a builder - #11586
Draft
joseph-isaacs wants to merge 3 commits into
Draft
joseph-isaacs wants to merge 3 commits into
joseph-isaacs wants to merge 3 commits into
Conversation
…ded to A chunked array appends each chunk to one builder. A chunk without its own append_to_builder executed into a buffer of its own, which was then copied into the builder. The default append_to_builder now runs the executor's loop with the builder as a sink: a fused chain that stands for the chunk is driven into the builder's uninitialized tail, and anything else is appended as before. BlockProgram::write_into runs a program into any slice of slots, and FusedChain::drive_into uses it for a primitive builder of the chain's dtype. Traces mark such drives with into=builder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Rebuilt the builder sink on the current executor: - The loop returns a fused chain that reaches the root undriven, instead of taking a sink parameter through every fusion function. execute_until drives it into a buffer; execute_and_append, the default append_to_builder, drives it into the builder, or into a buffer it then appends when the builder is not a primitive builder of its dtype. The executor knows nothing of builders. - drive_into reserves room for its rows first. Appending a fused FoR stack into a builder with no spare capacity panicked. - BlockProgram::write_into runs a program into any slice of slots. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part 3 of 3, stacked on #11542 (which is stacked on #11541). This draft checks that block fusion works with a builder as the sink: a fused chain that stands for a whole array can write straight into the builder that array is being appended to, for example each chunk of a
Chunkedarray.Without it, a chunk that has no
append_to_builderof its own is decoded into its own buffer, which is then copied into the builder.Changes
Executor: when a chain is extended all the way to the root, the loop returns it undriven, and each caller decides where it goes:
execute_untildrives it into a buffer, as before;ArrayRef::execute_and_append, used by the defaultappend_to_builder, drives it straight into the builder. If the builder isn't a primitive builder of the chain's dtype, it drives into a buffer and appends that.The executor itself never handles builders.
FusedChain::drive_into: reserves room first, writes the values into the builder's uninitialized tail throughBlockProgram::write_into, then appends the validity.Traces mark these drives with
into=builder.Encodings with their own
append_to_builder(BitPacked, Dict, Primitive, …) are unaffected.Tests
chunks_drive_into_the_builder(vortex-array), with the full trace asserted. Four chunks:chunks_without_fusion_append_level_wise.chunks_drive_into_the_builder(FastLanes):Chunkedof FoR stacks with nulls, patches and a sliced chunk. Every drive goes into the builder, and the result matches level-wise.drives_into_a_builder_without_capacity: a regression test for the review finding that appending a fused stack into an emptyPrimitiveBuilderpanicked.Checks
cargo test -p vortex-array -p vortex-fastlanes -p vortex-alp -p vortex-datetime-parts --libcargo clippy -p vortex-array -p vortex-fastlanes -p vortex-alp -p vortex-datetime-parts --all-targets --all-features -- -D warningscargo +nightly-2026-11-13 fmttypos, docs build, workspace-wide checks.🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U