Skip to content

reader: row-group snapshots pin batches outside the budget #79

Closed

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong.

  • RowGroupSnapshots (src/datafusion/src/cache/mod.rs:66-87) holds an Arc to each decoded batch, with no remove path and no charge.
  • The prefetch path snapshots every batch unconditionally (src/datafusion/src/reader/plantime/morselizer.rs:847). The reader snapshots on CacheFull (src/datafusion/src/reader/runtime/liquid_cache_reader.rs:435-436).
  • Eviction subtracts the entry from the budget while the snapshot still holds it, so the budget drops and real memory does not.
  • The headroom gates (morselizer.rs:695-707, :794-801) read the very counter that excludes snapshots, and they take no reservation. The first row group of each file is not gated (morselizer.rs:653-654); only the per-batch brake applies.
  • This is a concrete case of the "evicted while an Arc is still alive" candidate that Memory budget excludes hydrate/squeeze transients: tier reports at limit while the process holds 3x that #43 lists as unestablished.

Evidence.

Scenario Snapshot bytes Cache budget used Limit
Single prefetch 448 448 448
4 concurrent prefetches 11,648 7,168 7,168
Reader after CacheFull, 16 batches 3,584 0 1
flush_data while 64 snapshot entries are alive 7,168 0
  • For the 4 concurrent prefetches, snapshots alone are 1.62x the limit and snapshots plus cache are 2.6x.
  • The CacheFull row holds the whole row group, growing linearly until the stream yields None.
  • After flush_data, all 64 evicted arrays were still allocated.
  • First row group in a full cache: 896 snapshot bytes, so snapshots plus cache are 1.12x the limit.
  • Fixtures are 4-row Int32 batches, so real-workload scaling is inferred.

Tests. All run in a single-threaded runtime because the registry is thread-local, so use --test-threads=1.

Test File Command Flags
repro_prefetch_snapshots_exceed_memory_budget src/datafusion/src/reader/plantime/morselizer.rs:1943 cargo +1.96.0 test -p liquid-cache-datafusion --lib repro_prefetch_snapshots_exceed_memory_budget -- --nocapture --test-threads=1
repro_concurrent_prefetch_snapshots_exceed_memory_budget src/datafusion/src/reader/plantime/morselizer.rs:2055 cargo +1.96.0 test -p liquid-cache-datafusion --lib repro_concurrent_prefetch_snapshots_exceed_memory_budget -- --nocapture --test-threads=1
repro_cache_full_reader_parks_whole_row_group_in_snapshots src/datafusion/src/reader/plantime/morselizer.rs:2000 cargo +1.96.0 test -p liquid-cache-datafusion --lib repro_cache_full_reader_parks_whole_row_group_in_snapshots -- --nocapture --test-threads=1
audit_evicting_snapshotted_entries_releases_or_charges_them src/datafusion/src/reader/plantime/morselizer.rs:2155 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_evicting_snapshotted_entries_releases_or_charges_them -- --nocapture --test-threads=1
audit_first_row_group_prefetch_is_bounded_by_headroom src/datafusion/src/reader/plantime/morselizer.rs:2287 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_first_row_group_prefetch_is_bounded_by_headroom -- --nocapture --test-threads=1

The tests use the test-only snapshot registry.

Heads-up: the existing test snapshots_survive_eviction (morselizer.rs:1553) asserts this pinning as intended behaviour. A fix has to either charge snapshots or release them, and that test changes either way.

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    memoryMemory accounting or memory usage
    on Nov 17, 2026
  2. added a commit that references this issue on Nov 18, 2026
    466feec
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingmemoryMemory accounting or memory usage

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions