Skip to content

liquid: entries are charged without their per-node overhead #76

Closed

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong.

  • A liquid entry is charged self.array.nbytes() + size_of::<Self>() (src/core/src/liquid_array/array.rs:262-263). That leaves out the heap of the Vortex node tree; the doc comment at array.rs:93-100 already concedes this.
  • The overhead is roughly fixed per array, so the gap grows the better a column compresses and the smaller the batch.

Evidence.

  • real_heap <= charged * 1.10 + 256 failed for 24 of 41 fixtures.
Fixture Charged Real Ratio
Utf8 low-card(100) 7,736 16,771 2.17x
Utf8 low-card(16) 4,300 13,383 3.11x
Dict<u16, Utf8>(200) 9,966 19,049 1.91x
Utf8 3-distinct 2,136 4,836 2.26x
Constant, all-null or sequential 40–85 560–707 8–14.5x
  • The other 17 fixtures (Int32/Int64, Float64, Decimal128, random-step timestamps) are within 1.01–1.09x.
  • At cache level, live heap divided by the budget: Utf8LowCard 3.23x, TimestampUs 2.39x, DictUtf8 2.01x.
  • The low-card overhead is per-node heap, about 8.2 KB, mostly inferred to sit in the FSST node.
  • Projection, not measured: 9–28 GB per million well-compressed string or timestamp entries.

Tests.

Test File Command Flags
audit_liquid_charge_covers_real_heap src/core/src/liquid_array/size_audit.rs:470 cargo +1.96.0 test -p liquid-cache --lib audit_liquid_charge_covers_real_heap -- --nocapture
audit_report_where_uncharged_bytes_live src/core/src/liquid_array/size_audit.rs:571 cargo +1.96.0 test -p liquid-cache --lib audit_report_where_uncharged_bytes_live -- --nocapture M
audit_report_liquid_entries_heap_vs_budget src/core/tests/audit_cache_heap.rs:467 cargo +1.96.0 test -p liquid-cache --test audit_cache_heap audit_report_liquid_entries_heap_vs_budget -- --nocapture --test-threads=1 M
liquid_array_self_size src/core/tests/alloc_vs_budget.rs:525 cargo +1.96.0 test -p liquid-cache --test alloc_vs_budget liquid_array_self_size -- --nocapture --test-threads=1 M

Not inline: the test depends on a 228-line fixture table and a lib-wide counting global allocator.

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    memoryMemory accounting or memory usage
    on Nov 17, 2026
  2. added a commit that references this issue on Nov 18, 2026
    fb7f3a9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmemoryMemory accounting or memory usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions