Skip to content

cache: eviction deletes the disk copy of a victim flushed after selection #71

Open

Description

@zfarrell

Found by a memory-accounting audit of 8b3c813. The tests are on branch audit/memory-tests and fail on main today; check out that branch to run them. File and line links point at that commit.

What goes wrong.

  • evict_victim_inner reads the victim's slot when it runs (core.rs:845-850), not when the victim was chosen.
  • If the victim was flushed to disk after find_memory_victim returned it but before the eviction ran, the slot now holds a disk record. The eviction policy answers Remove for disk records (src/core/src/cache/policies/eviction.rs:76), and remove_disk_entry deletes the object that the flush just wrote (core.rs:886-888).
  • The cached data is lost. It is re-read from parquet, so this is not user data loss.
  • This is the same class as Disk-copy publication is not atomic with its ownership check #50 and Disk reclaim can delete an object a later write put at the same key #56: a decision made on a stale view of a slot.

Evidence. Victims 9810 and 9811 were flushed between selection and eviction. Afterwards both were unreadable, the store held 0 objects, and the index held 1 entry. The flush is injected by a test-only cache policy hook.

Tests.

Test File Command Flags
audit_a_memory_victim_flushed_before_its_eviction_keeps_its_disk_record src/core/src/cache/core.rs:3903 cargo +1.96.0 test -p liquid-cache --lib audit_a_memory_victim_flushed_before_its_eviction_keeps_its_disk_record -- --nocapture F (test policy HookAfterMemoryVictims)

Flags: P probabilistic (a pass after a fix only means the race wasn't hit), F interleaving forced by a test hook, L Linux only, D needs Docker, M measurement only, C control (passes by design).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions