Skip to content

reader: prefetch estimate under-counts decoded strings ~100x #80

Closed

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong. The headroom gate's estimate is estimated_bytes (src/datafusion/src/reader/runtime/morsel.rs:89-99), which sums parquet uncompressed_size(). That is the encoded page size before decoding, so dictionary- or RLE-encoded strings decode many times larger than the gate assumes.

Evidence. For a low-cardinality string column, the estimate was 21,968 bytes, but the prefetch pinned 2,228,720 bytes (0.0099x).

Tests.

Test File Command Flags
audit_prefetch_estimate_covers_decoded_dictionary_strings src/datafusion/src/reader/plantime/morselizer.rs:2207 cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_prefetch_estimate_covers_decoded_dictionary_strings -- --nocapture --test-threads=1 uses the test-only test_estimates hook

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingmemoryMemory accounting or memory usage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions