Skip to content

Push DataFusion runtime membership hints into Vortex scans - #11427

Closed
danking wants to merge 1 commit into
dk/df-hash-membership-expressionfrom
dk/df-hash-membership-pushdown
Closed

danking wants to merge 1 commit into
dk/df-hash-membership-expressionfrom
dk/df-hash-membership-pushdown

Conversation

@danking

@danking danking commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Superseded by #11431 and #11432, the reviewed generic DF-expression shim and mixed native/DF converter stack. Those replacements cover filters, projections, and partition-routing dynamic membership expressions.

Stacked on #11426; review this PR's single integration commit after the expression PR.

Teach the shared DataFusion converter to wrap HashTableLookupExpr, so Vortex scans can probe DF's existing join map before emitting rows. Ordinary conversion preserves exact DF errors. Positive dynamic-filter roots/conjuncts use a private advisory mode: evaluation failures retain the batch, including Arrow export failures from speculative evaluation of unused dictionary values. Advisory mode is never propagated through NOT or conditional branches.

A provided ExpressionConverter::try_convert_dynamic hook preserves infallible-only behavior for existing converters and documents conservative/no-new-value-error obligations for overrides. Existing snapshot refresh, generation tracking and join enforcement (PushedDown::No) remain. This does not add a generic DF-expression fallback or compile partition-routing CASE/hash expressions.

Eight focused cases pass: nullable single/composite hash membership, actual DF-created ArrayMap, invalid/ambiguous column binding, exact errors, positive-only advisory behavior, unused Dictionary<Int8,Utf8> export overflow, and a real Vortex source with projected renaming and two map generations. Command: cargo +nightly-2026-04-16 test --locked -p vortex-datafusion --lib hash_membership --no-default-features (8 passed, 0.05s). Formatting uses CI-pinned nightly-2026-09-10. No full test suite or throughput benchmark run.

@danking please review.

AI assistance: implementation and tests were prepared with Codex; commits were sent to the designated Codex review chat.

Signed-off-by: Daniel King <dan@spiraldb.com>
@robert3005 robert3005 added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Nov 12, 2026
@codspeed

codspeed Bot commented Nov 12, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 12.55%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 2 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

❌ 1 regressed benchmark
✅ 1920 untouched benchmarks
⏩ 50 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime scalar_subtract_neon 12 µs 13.7 µs -12.55%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing dk/df-hash-membership-pushdown (2856970) with dk/df-hash-membership-expression (3686cfc)

Open in CodSpeed

Footnotes

  1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

action/bench-sql Run SQL benchmarks without Vortex Compact on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants