Skip to content

fix(compressor): copy constant string and binary values out of the source buffer - #11382

Merged
robert3005 merged 3 commits into
vortex-data:developfrom
jackylee-ch:fix/compressor-constant-scalar-copy
Nov 11, 2026
Merged

robert3005 merged 3 commits into
vortex-data:developfrom
jackylee-ch:fix/compressor-constant-scalar-copy

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Summary

compress_constant builds the constant from execute_scalar, which for a VarBinView is a zero-copy slice of the chunk's data buffer, or of its views buffer for a value of at most 12 bytes. The constant is accepted because its nbytes is tiny, but it keeps the whole chunk alive, and BufferedStrategy weighs it by that tiny nbytes and buffers it until the end of the file. Writer memory grows with the row count.

Changes

compress_constant now copies the string and binary bytes of the scalar, including the elements of a fixed-size list, into a fresh buffer: one value per chunk. scalar_at stays zero-copy and BufferedStrategy is unchanged.

A throwaway write example (32,768-row batches, two constant string columns) peaks at 602 MiB on develop for 16M rows, 231 MiB for 4M rows. With this change, 16M rows peak at 221 MiB.

Tests

constant_does_not_alias_source_buffers fails on develop for an inlined string, an out-of-line string, a nullable string and binary values. The vortex-compressor tests and clippy pass.

AI assistance

Written with agentic AI assistance.

…urce buffer

compress_constant built the constant from execute_scalar, which for a
VarBinView is a zero-copy slice of the chunk's data or views buffer. The
constant was accepted because its nbytes is tiny, but it kept the whole
chunk alive, so the writer's memory grew with the row count. Copy the
string and binary bytes of the scalar, including those nested in a
fixed-size list, before building the ConstantArray.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@robert3005 robert3005 added the action/bench-compress Run only the compression benchmark on this PR label Nov 11, 2026
@robert3005 robert3005 assigned robert3005 and unassigned robert3005 Nov 11, 2026
@robert3005 robert3005 added the changelog/fix A bug fix label Nov 11, 2026
detach_value keeps the variant and shape of the value, so the scalar
needs no new validation.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 removed the action/bench-compress Run only the compression benchmark on this PR label Nov 11, 2026
@robert3005
robert3005 enabled auto-merge (squash) November 11, 2026 17:41
detach_scalar returns a Scalar and not a VortexResult, so the test must
not apply the ? operator to it.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants