Skip to content

cache: a read that cannot hydrate rewrites its object to disk #74

Open

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong.

  • A disk read hydrates by calling insert_inner (core.rs:917).
  • When the entry cannot fit in memory and the policy offers no memory victim, insert_inner writes the entry back to disk (core.rs:559-568). That rewrites the object the read just loaded.
  • Every such read costs a full disk write, a serialized copy, and WAL growth (cache: t4 file outgrows the disk budget #93).

Evidence.

  • With a 64-byte memory budget and a 1,024-row Int32 entry, two reads caused two writes; zero were expected.
  • Under production defaults with an incompressible array: one write per read, 5 of 5.
  • The original 0..1024 data does not reproduce under TranscodeEvict, because it compresses to about 40 bytes and stays in memory.

Tests.

Test File Command Flags
audit_a_read_that_cannot_hydrate_writes_nothing src/core/src/cache/core.rs:4138 cargo +1.96.0 test -p liquid-cache --lib audit_a_read_that_cannot_hydrate_writes_nothing Inline
audit_prod_unhydratable_read_writes_nothing_sequential src/core/src/cache/core.rs:3106 cargo +1.96.0 test -p liquid-cache --lib audit_prod_unhydratable_read_writes_nothing_sequential -- --nocapture
audit_prod_unhydratable_read_writes_nothing_concurrent src/core/src/cache/core.rs:3115 cargo +1.96.0 test -p liquid-cache --lib audit_prod_unhydratable_read_writes_nothing_concurrent -- --nocapture

Inline test:

    #[tokio::test]
    async fn audit_a_read_that_cannot_hydrate_writes_nothing() {
        let directory = tempfile::tempdir().unwrap();
        let store = t4::mount(directory.path().join("cache.t4")).await.unwrap();
        let cache = LiquidCacheBuilder::new()
            .with_max_memory_bytes(64)
            .with_max_disk_bytes(1 << 20)
            .with_eviction_policy(Box::new(crate::cache::Evict))
            .with_cache_policy(Box::new(LiquidPolicy::new()))
            .with_hydration_policy(Box::new(AlwaysHydrate::new()))
            .with_store(store.clone())
            .build()
            .await;
        let entry = EntryID::from(1030usize);
        let array: ArrayRef = Arc::new(Int32Array::from_iter_values(0..1024));
        cache.insert(entry, 0, array.clone()).await.unwrap();
        assert!(matches!(
            cache.index().get(&entry).unwrap().entry,
            CacheEntry::DiskArrow { .. }
        ));
        let inserting = cache.observer().runtime_snapshot().write_io_count;
        assert_eq!(inserting, 1, "the insert spilled the entry once");

        for _ in 0..2 {
            let read = cache.get(&entry, 0).await.expect("present");
            assert_eq!(read.as_ref(), array.as_ref());
        }

        assert!(matches!(
            cache.index().get(&entry).unwrap().entry,
            CacheEntry::DiskArrow { .. }
        ));
        assert_disk_matches_index(&cache, &store).await;
        assert_eq!(
            cache.observer().runtime_snapshot().write_io_count,
            0,
            "reads rewrote the object they had just read"
        );
    }

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

  1. added
    bugSomething isn't working
    performanceWasted work, over-eviction or excess IO
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformanceWasted work, over-eviction or excess IO

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions