Skip to content

datafusion: shared dictionary values charged once per batch #83

Open

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong. Batches read from a dictionary-typed parquet column share one values array. Each cache entry is charged the full dictionary through get_array_memory_size() (src/core/src/cache/cached_batch.rs:63), so the cache over-counts and evicts more than it needs to. This is the same counting mechanism as #75, but across entries rather than within one.

Evidence.

  • 32 batches really holding 1,212,420 bytes (34 distinct buffers) were charged 34,742,656 bytes: 28.66x.
  • The allocator run saw the same effect on slices of one buffer: 64 disjoint slices charged 64x.

Tests.

Test File Command Flags
audit_shared_dictionary_is_charged_once src/datafusion/src/reader/plantime/morselizer.rs:2385 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_shared_dictionary_is_charged_once -- --nocapture --test-threads=1
alloc_vs_budget (SharedBuffer and SlicedWindow rows) src/core/tests/alloc_vs_budget.rs:464 see #43 M

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    performanceWasted work, over-eviction or excess IO
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformanceWasted work, over-eviction or excess IO

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions