Skip to content

liquid: decode copies misaligned buffers and keeps the block #77

Closed

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong.

  • When a buffer lands at a misaligned offset in the serialized block, decoding makes an aligned copy. Vortex's ensure_aligned (vortex-array/src/serde.rs:692,703) is the inferred source.
  • The other buffers still pin the whole block, so the copied bytes are held twice, and nbytes() counts them once.
  • Whether an entry pays depends on its byte layout: the same Binary data with nulls copied nothing.

Evidence. bytes in separate allocations == 0 failed for 21 of 41 fixtures, 8 of them material:

Fixture Bytes copied Charged → real
Binary random 16 B (varbinview.views) 131,072 262,184 → 394,588 (1.51x)
Timestamp ramp (bitpacked.packed) 20,520 20,587 → 48,123
FSST strings (bitpacked.packed) 19.5–21 KB each

This also resolves the earlier timestamp discrepancy (27.5 KB vs 2.5 KB uncharged): only one of the two fixtures hit the copy.

Tests.

Test File Command Flags
audit_decode_copies_no_buffer_out_of_the_serialized_block src/core/src/liquid_array/size_audit.rs:515 cargo +1.96.0 test -p liquid-cache --lib audit_decode_copies_no_buffer_out_of_the_serialized_block -- --nocapture

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    memoryMemory accounting or memory usage
    on Nov 17, 2026
  2. added a commit that references this issue on Nov 18, 2026
    27388fe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmemoryMemory accounting or memory usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions