Skip to content

Add a Vortex expression for DataFusion runtime membership - #11426

Closed
danking wants to merge 1 commit into
developfrom
dk/df-hash-membership-expression
Closed

danking wants to merge 1 commit into
developfrom
dk/df-hash-membership-expression

Conversation

@danking

@danking danking commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Superseded by #11431 and #11432, the reviewed generic DF-expression shim and mixed native/DF converter stack. Those replacements cover filters, projections, and partition-routing dynamic membership expressions.

DataFusion exposes join membership through a runtime HashTableLookupExpr backed by a shared hash table or dense array map. Add a process-local Vortex expression that delegates to that lookup, preserving its map, hash seed, composite keys, null behavior and errors. Its result is non-nullable Boolean; hash-backed membership retains DF's advisory collision semantics rather than claiming SQL IN semantics.

The wrapper exports only referenced raw columns to Arrow, validates and rebinds their indices, preserves Arrow fields, and declines volatile, asynchronous or live-dynamic keys. It is conservatively fallible and cannot be serialized. No table-provider conversion changes are included; those follow in a stacked PR.

Added focused coverage for nullable single/composite hash keys, exact key-evaluation errors, duplicate/crossed column references, and an actual DF-created ArrayMap with null/out-of-range probes. The first five cases pass as part of the eight-case stacked check: cargo +nightly-2026-04-16 test --locked -p vortex-datafusion --lib hash_membership --no-default-features (8 passed, 0.05s). Formatting uses the CI-pinned nightly-2026-09-10. No full suite or performance benchmark run.

@danking please review.

AI assistance: implementation and tests were prepared with Codex and reviewed in the designated Codex review chat.

Stacked integration: #11427.

Signed-off-by: Daniel King <dan@spiraldb.com>
@codspeed

codspeed Bot commented Nov 12, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 12.49%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 3 improved benchmarks
✅ 1918 untouched benchmarks
⏩ 50 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime scalar_subtract_neon 13.8 µs 12 µs +14.98%
⚡ Walltime decode_chunked_neon[i64, 524288] 15.4 µs 13.7 µs +12.45%
⚡ Walltime add_shapes_neon[(32768, PerRowPerRow)] 18.8 µs 17.1 µs +10.09%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing dk/df-hash-membership-expression (3686cfc) with develop (d81bd4c)

Open in CodSpeed

Footnotes

  1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant