You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
liquid: entries are charged without their per-node overhead #76
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong.
A liquid entry is charged self.array.nbytes() + size_of::<Self>() (src/core/src/liquid_array/array.rs:262-263). That leaves out the heap of the Vortex node tree; the doc comment at array.rs:93-100 already concedes this.
The overhead is roughly fixed per array, so the gap grows the better a column compresses and the smaller the batch.
Evidence.
real_heap <= charged * 1.10 + 256 failed for 24 of 41 fixtures.
Fixture
Charged
Real
Ratio
Utf8 low-card(100)
7,736
16,771
2.17x
Utf8 low-card(16)
4,300
13,383
3.11x
Dict<u16, Utf8>(200)
9,966
19,049
1.91x
Utf8 3-distinct
2,136
4,836
2.26x
Constant, all-null or sequential
40–85
560–707
8–14.5x
The other 17 fixtures (Int32/Int64, Float64, Decimal128, random-step timestamps) are within 1.01–1.09x.
At cache level, live heap divided by the budget: Utf8LowCard 3.23x, TimestampUs 2.39x, DictUtf8 2.01x.
The low-card overhead is per-node heap, about 8.2 KB, mostly inferred to sit in the FSST node.
Projection, not measured: 9–28 GB per million well-compressed string or timestamp entries.
Not inline: the test depends on a 228-line fixture table and a lib-wide counting global allocator.
Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Activity
fix(liquid): charge entries for their per-node heap