You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Confirm stale-file semantics: the cache is keyed by path with no invalidation #36
Part of #22. A question to answer, not necessarily code to write.
The cache is keyed by file path. register_or_get_file(file_path: String, full_file_schema: SchemaRef) (src/datafusion/src/cache/mod.rs:326) takes the path and nothing else — no mtime, no size, no etag, no generation number. There is no invalidation path.
So if a file is rewritten in place at the same path, a warm cache serves the old contents indefinitely. No oracle in #22 would catch this, because every harness arm writes each file once.
Why this is worth resolving now
This is most likely immutable-files-by-design, and for an immutable-file layout it is the correct and fastest thing to do. But this crate is embedded by runtimedb, which compacts and rewrites managed tables. If a compaction can ever reuse a path, this is a live wrong-results bug in production rather than a hypothetical.
What to do
Confirm whether any embedder can rewrite a path the cache has seen. Check runtimedb's compaction and its DuckLake file naming specifically — whether rewritten files always get fresh paths.
If paths are guaranteed unique per version: document the invariant on register_or_get_file so the assumption is explicit and a future change to the naming scheme trips over it.
If they are not: this becomes a correctness bug and needs its own issue and a real fix — key on path plus a version discriminator, or expose an explicit invalidation call.
Acceptance
Either a documented invariant with the reasoning recorded, or a new issue tracking the bug. Not an unexamined assumption.
Activity