Repository navigation
Conversation
Signed-off-by: Daniel King <dan@spiraldb.com>
Merging this PR will not alter performance2 warnings
Comparing Footnotes
|
Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
|
@claude review |
|
Claude finished @danking's task in 1m 20s —— View job Review of #11431: typed DataFusion expression shim
The design is solid. It keeps decimal outputs in DF, rejects live, volatile and async expressions, checks the output type at runtime with 1. Advisory mode swallows input execution errors, not just DF evaluation errors (worth fixing)
So a corrupt segment or a type mismatch in the shim would silently disable the filter instead of failing the query. The PR description promises "retains rows on optional-filter evaluation failure". I'd narrow the fallback to that: convert the inputs and build the batch with The There is also no direct test of advisory behavior in this PR. Nothing checks that a DF error becomes all- 2. Ambiguous column names give inconsistent results: an error in one path,
|
DataFusion fragments that Vortex cannot execute natively can now run inside a Vortex expression tree through a typed, process-local
df_expressionScalarFn. The wrapper preserves DF result types, nulls, Arrow field metadata, runtime join-map state, and evaluation errors; it exports referenced inputs and cannot be serialized. Callbacks containing higher-order functions retain the full original input layout to preserve parameter indices.The public API accepts synchronous, non-volatile, static physical expressions. Live filters are handled through snapshots in the stacked integration. Unsupported Vortex result types, including decimal outputs whose Arrow coefficients need not fit the declared precision, remain in DF; decimal intermediates inside a representable result are allowed. An internal Boolean-only advisory mode retains rows on optional-filter evaluation failure; exact evaluation always returns errors.
Focused tests cover nullable single/composite hash lookups, a real DF-created dense ArrayMap, invalid column bindings, typed projections, scalar row counts, wrapping arithmetic, rejected volatile/live expressions, and a registered Boolean extension's inferred dtype and executed values. These eight shim cases pass within the stacked expression check:
cargo +nightly-2026-04-16 test --locked -p vortex-datafusion --lib df_expression --no-default-features(16 cases total, 0.06s). Additional stacked tests check UDF-visible fields/scalar shape, lambda parameter bindings, and DF-owned CASE projection rewrites. Formatting uses CI-pinned nightly-2026-09-10.Supersedes the membership-only #11426. @danking please review.
AI assistance: prepared with Codex and reviewed in the designated Codex review chat.
Stacked converter integration: #11432.