Repository navigation
Conversation
Signed-off-by: Daniel King <dan@spiraldb.com>
Merging this PR will degrade performance by 12.55%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Superseded by #11431 and #11432, the reviewed generic DF-expression shim and mixed native/DF converter stack. Those replacements cover filters, projections, and partition-routing dynamic membership expressions.
Stacked on #11426; review this PR's single integration commit after the expression PR.
Teach the shared DataFusion converter to wrap
HashTableLookupExpr, so Vortex scans can probe DF's existing join map before emitting rows. Ordinary conversion preserves exact DF errors. Positive dynamic-filter roots/conjuncts use a private advisory mode: evaluation failures retain the batch, including Arrow export failures from speculative evaluation of unused dictionary values. Advisory mode is never propagated through NOT or conditional branches.A provided
ExpressionConverter::try_convert_dynamichook preserves infallible-only behavior for existing converters and documents conservative/no-new-value-error obligations for overrides. Existing snapshot refresh, generation tracking and join enforcement (PushedDown::No) remain. This does not add a generic DF-expression fallback or compile partition-routing CASE/hash expressions.Eight focused cases pass: nullable single/composite hash membership, actual DF-created ArrayMap, invalid/ambiguous column binding, exact errors, positive-only advisory behavior, unused Dictionary<Int8,Utf8> export overflow, and a real Vortex source with projected renaming and two map generations. Command:
cargo +nightly-2026-04-16 test --locked -p vortex-datafusion --lib hash_membership --no-default-features(8 passed, 0.05s). Formatting uses CI-pinned nightly-2026-09-10. No full test suite or throughput benchmark run.@danking please review.
AI assistance: implementation and tests were prepared with Codex; commits were sent to the designated Codex review chat.