Skip to content

cache: two caches can mount one t4 file #96

Open

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong.

  • The local builder's default cache_dir is std::env::temp_dir() (src/datafusion-local/src/lib.rs:87), and the file name is fixed (lib.rs:196). Any two default builds, in one process or in two processes sharing a temp directory, mount the same file.
  • t4 takes no lock, so each mount allocates pages independently and their writes overwrite each other's objects. That mechanism is inferred.
  • Disk reads can return another entry's bytes (t4 slot reuse) #63 is a different root cause with a similar symptom: another entry's bytes inside an object.

Evidence.

  • A second t4::mount of the same path succeeded.
  • After B spilled, 3,139 of A's 4,560 object bytes were replaced by the start of B's object. A's next read returned a miss, not wrong data.

Tests.

Test File Command Flags
audit_two_caches_over_one_cache_file_are_refused_or_share_state src/core/src/cache/builders_audit_tests.rs:106 cargo +1.96.0 test -p liquid-cache --lib audit_two_caches_over_one_cache_file_are_refused_or_share_state -- --nocapture
audit_two_default_builds_do_not_mount_the_same_cache_file src/datafusion-local/src/audit_tests.rs:127 cargo +1.96.0 test -p liquid-cache-datafusion-local --lib audit_two_default_builds_do_not_mount_the_same_cache_file -- --nocapture Inline; mounts the real $TMPDIR/liquid_cache.t4

Inline test (inside the local crate, because cache_dir is private):

#[tokio::test]
async fn audit_two_default_builds_do_not_mount_the_same_cache_file() {
    let a = LiquidCacheLocalBuilder::new();
    let b = LiquidCacheLocalBuilder::new();
    let (a_file, b_file) = (
        a.cache_dir.join("liquid_cache.t4"),
        b.cache_dir.join("liquid_cache.t4"),
    );
    let (_a_ctx, a_cache) = a.build(SessionConfig::new()).await.unwrap();
    let Ok((_b_ctx, b_cache)) = b.build(SessionConfig::new()).await else {
        return;
    };
    let shared = std::sync::Arc::ptr_eq(a_cache.storage(), b_cache.storage());
    assert!(
        a_file != b_file || shared,
        "two default builds each mounted {} as an independent cache (shared = {shared})",
        a_file.display()
    );
}

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions