Repository navigation
fix(liquid): charge entries for the memory they hold - #101
Merged
Merged
Conversation
zfarrell
marked this pull request as ready for review
November 17, 2026 20:35
zfarrell
requested review from
shefeek-jinnah
and removed request for
a team
November 17, 2026 20:35
Liquid arrays were serialized as Vortex IPC messages, which place buffers with no padding. Decoding copied every buffer that landed at an offset its alignment rejects, while the rest kept the whole block alive, so the copied bytes were held twice and charged once: up to 1.51x the charge for random binary. Write the array segment with Vortex's zero-copy padding instead, in an allocation aligned for its largest buffer, and realign a block handed to from_bytes once if its allocation is not aligned enough. The format version moves to 2; the disk tier only reads blocks this process wrote. Fixes #77
A Liquid entry was charged nbytes(), the sum of its buffer lengths. That leaves out the serialized block's framing and padding, each Vortex node's allocation and statistics, and the FSST symbol table the decoder re-pads to its full 256 entries. For well-compressed columns that was most of the entry: low-cardinality strings held 2.2-2.4x their charge, constant and sequential columns 15x. Compute the charge once at decode: the block if any buffer still points into it, every buffer that does not, and a measured per-node and per-FSST-node overhead. A counting-allocator test, scoped to its own test binary, holds the charge within 10% + 256 bytes of the real heap in both directions across 28 column shapes. Two tests sized their budget from the old under-charge: the 128-row string OR test now uses 1024 rows so its Liquid form is still the smaller, and the URL snapshot keeps one fewer entry in memory. Fixes #76
FSST evaluates = and != by compressing the literal, and memoizes the 160 KiB compressor it builds in the symbol table that every slice and filter of the node shares. The first such predicate on a cached entry grew it for the rest of its life, uncharged: 1.8x the whole charge of a 16-character hex column. Give the comparison a private copy of each FSST node's symbol table, so the compressor is freed with the predicate's array. Pre-charging every FSST entry for a compressor most never build would over-charge them, and re-charging after growth needs a hook from the array into the budget. Rebuilding the compressor costs 15-30 us per 8192-row batch. Fixes #78
- The predicate test asserts its columns are FSST-compressed, so an encoding change cannot make it pass without exercising FSST. - Small string batches must be charged at least the heap they hold. A full FSST symbol table is over-charged by about 2 KB, which costs cache space rather than memory. - Note how to re-derive NODE_HEAP and FSST_SYMBOL_TABLE_HEAP after a Vortex upgrade. - Regenerate the url_selection_and_ordering snapshot after the rebase. Claude-Session: https://claude.ai/code/session_01LbJrgYNLF7FxwMy5VSHXar
anoop-narang
force-pushed
the
fix/liquid-memory-charge
branch
from
November 18, 2026 16:24
c7555e6 to
1b7a6f1
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
📊 Benchmark ComparisonCurrent:
Compared Liquid vs DataFusionDefault on the same runner |
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Liquid entries are now charged within about 10% of the heap they hold.
NODE_HEAP) and per FSST symbol table (FSST_SYMBOL_TABLE_HEAP), measured with a counting allocator.tests/liquid_array_memory.rschecks charges against real heap and fails if the allowances drift, e.g. after a Vortex upgrade; the constants' doc says how to re-derive them.=/!=no longer grow the entry (liquid: FSST = and != grow an entry by 160 KiB uncharged #78). They evaluate on a copy with its own symbol table, so the ~160 KiB compressor Vortex builds for the literal is dropped with the copy. Trade-off: the compressor is rebuilt on each evaluation, a few µs per batch, instead of being kept per entry uncharged.Small batches of high-entropy strings are over-charged by about 2 KB (a full symbol table holds less than the allowance). The error is deliberately on the safe side: over-charging costs cache space, under-charging overruns memory.
Closes #77
Closes #76
Closes #78
https://claude.ai/code/session_01LbJrgYNLF7FxwMy5VSHXar