Repository navigation
Description
Summary
A disk read can return the bytes of a different cache entry. The identity trailer added in datafusion-contrib#526 does not catch it when both entries carry the same identity, so the wrong bytes are hydrated into the entry's memory record and served from then on.
The root cause is in t4 0.1.9, the store the disk tier uses. It is present on fork main and on older fork revisions that use the same t4 version.
Mechanism
t4::Store::getlooks up where the key's value lives, then reads that location after an.await.- A concurrent
putorremoveof the same key frees the old location immediately (release_value_space). - A
putof a different key can be given that freed space before the pending read runs. - The read returns the other key's bytes.
At the cache level, read_disk_bytes checks only the identity trailer. The trailer stores the identity of the file the entry was cached for, not the entry key. So when two entries share an identity (for example two columns, or two batches, of one file), the other entry's bytes pass the check.
Reproduction (t4 alone, no liquid-cache)
Two writer threads alternate put/remove over 4 keys, with each key's mutations serialized by a per-key mutex. Two reader threads get the same keys and check that every byte equals the key's own value:
#[test]
fn t4_get_returns_another_keys_bytes() {
use std::sync::{Arc, Mutex};
let dir = tempfile::tempdir().unwrap();
let store = tokio_test::block_on(t4::mount(dir.path().join("s.t4"))).unwrap();
let locks: Arc<Vec<Mutex<()>>> = Arc::new((0..4).map(|_| Mutex::new(())).collect());
let value = |k: u8| vec![k + 1; 1500];
let mut threads = Vec::new();
for t in 0..2u8 {
let (s, locks) = (store.clone(), locks.clone());
threads.push(std::thread::spawn(move || for i in 0..3000u32 {
let k = ((i + t as u32 * 7) % 4) as u8;
let _g = locks[k as usize].lock().unwrap();
tokio_test::block_on(async {
if (i / 4 + t as u32) % 2 == 0 { s.put(vec![k], value(k)).await.unwrap(); }
else { s.remove(&[k]).await.unwrap(); }
});
}));
}
for _ in 0..2 {
let s = store.clone();
threads.push(std::thread::spawn(move || for i in 0..6000u32 {
let k = (i % 4) as u8;
if let Ok(v) = tokio_test::block_on(s.get(&[k])) {
assert!(v.iter().all(|b| *b == k + 1), "key {k} read bytes of key {}", v[0] - 1);
}
}));
}
for t in threads { t.join().unwrap(); }
}Observed:
- With per-key serialization and 4 keys: a wrong read in 30 of 30 runs.
- Without it (two writers per key, 2 keys): in 14 of 50 runs.
- Every wrong read was transient. The store reads correctly once writes stop, in 0 of 30 runs.
A concurrent hydrate/flush/insert probe on the cache itself (4 keys, 1 GiB disk cap, data checked on every read) served another entry's data in 766 of 1000 runs on main. Single-threaded, it is clean.
Possible fixes
- In
t4: don't hand out a freed location while a read of it may be in flight (defer the release until pending reads finish, or use generation-checked reads). This fixes the cause. - In the cache (mitigation): write the entry key into the trailer next to the identity, and check both on read. A mismatch then becomes a miss instead of wrong data. It is small and local, but it costs one key's worth of bytes per object.
Both together would be the most robust.
This issue was investigated with AI assistance (Claude Code).
Activity
zfarrell commentedon Nov 13, 2026
Wow! Great find. that is extremely concerning. do you think we should disable liquid cache disk until this is fixed?