Repository navigation
fix(compressor): copy constant string and binary values out of the source buffer - #11382
Merged
robert3005 merged 3 commits intoNov 11, 2026
Merged
robert3005 merged 3 commits into
robert3005 merged 3 commits into
Conversation
…urce buffer compress_constant built the constant from execute_scalar, which for a VarBinView is a zero-copy slice of the chunk's data or views buffer. The constant was accepted because its nbytes is tiny, but it kept the whole chunk alive, so the writer's memory grew with the row count. Copy the string and binary bytes of the scalar, including those nested in a fixed-size list, before building the ConstantArray. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
robert3005
reviewed
Nov 11, 2026
robert3005
approved these changes
Nov 11, 2026
detach_value keeps the variant and shape of the value, so the scalar needs no new validation. Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005
enabled auto-merge (squash)
November 11, 2026 17:41
detach_scalar returns a Scalar and not a VortexResult, so the test must not apply the ? operator to it. Signed-off-by: Robert Kruszewski <github@robertk.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
compress_constantbuilds the constant fromexecute_scalar, which for aVarBinViewis a zero-copy slice of the chunk's data buffer, or of its views buffer for a value of at most 12 bytes. The constant is accepted because itsnbytesis tiny, but it keeps the whole chunk alive, andBufferedStrategyweighs it by that tinynbytesand buffers it until the end of the file. Writer memory grows with the row count.Changes
compress_constantnow copies the string and binary bytes of the scalar, including the elements of a fixed-size list, into a fresh buffer: one value per chunk.scalar_atstays zero-copy andBufferedStrategyis unchanged.A throwaway write example (32,768-row batches, two constant string columns) peaks at 602 MiB on
developfor 16M rows, 231 MiB for 4M rows. With this change, 16M rows peak at 221 MiB.Tests
constant_does_not_alias_source_buffersfails ondevelopfor an inlined string, an out-of-line string, a nullable string and binary values. Thevortex-compressortests and clippy pass.AI assistance
Written with agentic AI assistance.