You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
local: builder runs DataFusion on an unbounded pool #86
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong.LiquidCacheLocalBuilder::build takes only a SessionConfig and builds the context at src/datafusion-local/src/lib.rs:236. The result runs on UnboundedMemoryPool, and a caller has no way to pass a RuntimeEnv that carries a bounded one.
Evidence. With the cache configured to 1 GiB, the pool was UnboundedMemoryPool { used: 0 } and memory_limit() returned Infinite.
cargo +1.96.0 test -p liquid-cache-datafusion-local --lib audit_local_context_has_a_bounded_memory_pool -- --nocapture
Inline
Inline test (needs use datafusion::execution::memory_pool::MemoryLimit; use datafusion::prelude::SessionConfig; const GIB: usize = 1 << 30;):
#[tokio::test]asyncfnaudit_local_context_has_a_bounded_memory_pool(){let dir = tempfile::tempdir().unwrap();let(ctx, _cache) = LiquidCacheLocalBuilder::new().with_cache_dir(dir.path().to_path_buf()).with_max_memory_bytes(GIB).build(SessionConfig::new()).await.unwrap();let pool = ctx.runtime_env().memory_pool.clone();assert!(
matches!(pool.memory_limit(),MemoryLimit::Finite(_)),"the SessionContext built by LiquidCacheLocalBuilder runs on an unbounded \ DataFusion pool: {pool:?}");}
(The branch's diagnostic println! is trimmed here.)
Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Activity