Repository navigation
Conversation
Signed-off-by: Daniel King <dan@spiraldb.com>
Merging this PR will improve performance by 12.49%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Signed-off-by: Daniel King <dan@spiraldb.com>
Merging this PR will improve performance by 12.49%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Superseded by #11431 and #11432, the reviewed generic DF-expression shim and mixed native/DF converter stack. Those replacements cover filters, projections, and partition-routing dynamic membership expressions.
DataFusion exposes join membership through a runtime
HashTableLookupExprbacked by a shared hash table or dense array map. Add a process-local Vortex expression that delegates to that lookup, preserving its map, hash seed, composite keys, null behavior and errors. Its result is non-nullable Boolean; hash-backed membership retains DF's advisory collision semantics rather than claiming SQLINsemantics.The wrapper exports only referenced raw columns to Arrow, validates and rebinds their indices, preserves Arrow fields, and declines volatile, asynchronous or live-dynamic keys. It is conservatively fallible and cannot be serialized. No table-provider conversion changes are included; those follow in a stacked PR.
Added focused coverage for nullable single/composite hash keys, exact key-evaluation errors, duplicate/crossed column references, and an actual DF-created ArrayMap with null/out-of-range probes. The first five cases pass as part of the eight-case stacked check:
cargo +nightly-2026-04-16 test --locked -p vortex-datafusion --lib hash_membership --no-default-features(8 passed, 0.05s). Formatting uses the CI-pinned nightly-2026-09-10. No full suite or performance benchmark run.@danking please review.
AI assistance: implementation and tests were prepared with Codex and reviewed in the designated Codex review chat.
Stacked integration: #11427.