You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong. The headroom gate's estimate is estimated_bytes (src/datafusion/src/reader/runtime/morsel.rs:89-99), which sums parquet uncompressed_size(). That is the encoded page size before decoding, so dictionary- or RLE-encoded strings decode many times larger than the gate assumes.
Evidence. For a low-cardinality string column, the estimate was 21,968 bytes, but the prefetch pinned 2,228,720 bytes (0.0099x).
cargo +1.96.0 test -p liquid-cache-datafusion --lib audit_prefetch_estimate_covers_decoded_dictionary_strings -- --nocapture --test-threads=1
uses the test-only test_estimates hook
Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Activity
fix(reader): size prefetch by decoded bytes (#99)