Repository navigation
Open
Labels
bugSomething isn't workingSomething isn't workingmemoryMemory accounting or memory usageMemory accounting or memory usage
Description
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong.
LiquidCacheService::newpassesmax_memory_bytes.unwrap_or(usize::MAX)for memory and a hard-codedusize::MAXfor disk (src/datafusion-server/src/service.rs:67-68). With no flag set, the server has no limits, and disk stays unlimited even when memory is configured./cache_info(src/datafusion-server/src/admin_server/handlers.rs:129-142) reports thatusize::MAXas the memory limit and has nomax_disk_bytesfield.
Evidence. Both memory and disk came out as 18,446,744,073,709,551,615 with no flag set. Disk was the same with memory set to 1 GiB, and /cache_info reported the same value.
Tests.
| Test | File | Command | Flags |
|---|---|---|---|
audit_service_default_budgets_are_finite |
src/datafusion-server/src/tests/audit_config.rs:20 |
cargo +1.96.0 test -p liquid-cache-datafusion-server --lib audit_service_default_budgets_are_finite -- --nocapture |
Inline |
audit_service_disk_budget_is_finite_when_memory_is_configured |
src/datafusion-server/src/tests/audit_config.rs:54 |
cargo +1.96.0 test -p liquid-cache-datafusion-server --lib audit_service_disk_budget_is_finite_when_memory_is_configured -- --nocapture |
|
audit_cache_info_reports_a_real_memory_limit |
src/datafusion-server/src/admin_server/handlers_audit_tests.rs:90 |
cargo +1.96.0 test -p liquid-cache-datafusion-server --lib audit_cache_info_reports_a_real_memory_limit -- --nocapture |
|
audit_system_info_reports_the_memory_the_server_may_use |
src/datafusion-server/src/admin_server/handlers_audit_tests.rs:113 |
cargo +1.96.0 test -p liquid-cache-datafusion-server --lib audit_system_info_reports_the_memory_the_server_may_use -- --nocapture |
M on macOS; the assertion is Linux-gated and was not run |
Inline test:
#[tokio::test]
async fn audit_service_default_budgets_are_finite() {
let dir = tempfile::tempdir().unwrap();
let service = LiquidCacheService::new(
LiquidCacheService::context().unwrap(),
None,
Some(dir.path().to_path_buf()),
Box::new(LiquidPolicy::new()),
Box::new(TranscodeEvict),
Box::new(AlwaysHydrate::new()),
)
.await
.unwrap();
let memory = service.cache().max_memory_bytes();
let disk = service.cache().max_disk_bytes();
let mut unbounded = Vec::new();
if memory == usize::MAX {
unbounded.push("max_memory_bytes");
}
if disk == usize::MAX {
unbounded.push("max_disk_bytes");
}
assert!(
unbounded.is_empty(),
"LiquidCacheService::new(ctx, None, ..) leaves {unbounded:?} at usize::MAX \
(memory = {memory}, disk = {disk})"
);
}Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingmemoryMemory accounting or memory usageMemory accounting or memory usage
Activity