You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
datafusion: shared dictionary values charged once per batch #83
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong. Batches read from a dictionary-typed parquet column share one values array. Each cache entry is charged the full dictionary through get_array_memory_size() (src/core/src/cache/cached_batch.rs:63), so the cache over-counts and evicts more than it needs to. This is the same counting mechanism as #75, but across entries rather than within one.
Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Activity