Repository navigation
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.
What goes wrong.
- The local builder's default
cache_dirisstd::env::temp_dir()(src/datafusion-local/src/lib.rs:87), and the file name is fixed (lib.rs:196). Any two default builds, in one process or in two processes sharing a temp directory, mount the same file. - t4 takes no lock, so each mount allocates pages independently and their writes overwrite each other's objects. That mechanism is inferred.
- Disk reads can return another entry's bytes (t4 slot reuse) #63 is a different root cause with a similar symptom: another entry's bytes inside an object.
Evidence.
- A second
t4::mountof the same path succeeded. - After B spilled, 3,139 of A's 4,560 object bytes were replaced by the start of B's object. A's next read returned a miss, not wrong data.
Tests.
| Test | File | Command | Flags |
|---|---|---|---|
audit_two_caches_over_one_cache_file_are_refused_or_share_state |
src/core/src/cache/builders_audit_tests.rs:106 |
cargo +1.96.0 test -p liquid-cache --lib audit_two_caches_over_one_cache_file_are_refused_or_share_state -- --nocapture |
|
audit_two_default_builds_do_not_mount_the_same_cache_file |
src/datafusion-local/src/audit_tests.rs:127 |
cargo +1.96.0 test -p liquid-cache-datafusion-local --lib audit_two_default_builds_do_not_mount_the_same_cache_file -- --nocapture |
Inline; mounts the real $TMPDIR/liquid_cache.t4 |
Inline test (inside the local crate, because cache_dir is private):
#[tokio::test]
async fn audit_two_default_builds_do_not_mount_the_same_cache_file() {
let a = LiquidCacheLocalBuilder::new();
let b = LiquidCacheLocalBuilder::new();
let (a_file, b_file) = (
a.cache_dir.join("liquid_cache.t4"),
b.cache_dir.join("liquid_cache.t4"),
);
let (_a_ctx, a_cache) = a.build(SessionConfig::new()).await.unwrap();
let Ok((_b_ctx, b_cache)) = b.build(SessionConfig::new()).await else {
return;
};
let shared = std::sync::Arc::ptr_eq(a_cache.storage(), b_cache.storage());
assert!(
a_file != b_file || shared,
"two default builds each mounted {} as an independent cache (shared = {shared})",
a_file.display()
);
}Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity