Skip to content

Add a typed DataFusion expression shim for Vortex - #11431

Open
danking wants to merge 3 commits into
developfrom
dk/df-expression-shim
Open

danking wants to merge 3 commits into
developfrom
dk/df-expression-shim

Conversation

@danking

@danking danking commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

DataFusion fragments that Vortex cannot execute natively can now run inside a Vortex expression tree through a typed, process-local df_expression ScalarFn. The wrapper preserves DF result types, nulls, Arrow field metadata, runtime join-map state, and evaluation errors; it exports referenced inputs and cannot be serialized. Callbacks containing higher-order functions retain the full original input layout to preserve parameter indices.

The public API accepts synchronous, non-volatile, static physical expressions. Live filters are handled through snapshots in the stacked integration. Unsupported Vortex result types, including decimal outputs whose Arrow coefficients need not fit the declared precision, remain in DF; decimal intermediates inside a representable result are allowed. An internal Boolean-only advisory mode retains rows on optional-filter evaluation failure; exact evaluation always returns errors.

Focused tests cover nullable single/composite hash lookups, a real DF-created dense ArrayMap, invalid column bindings, typed projections, scalar row counts, wrapping arithmetic, rejected volatile/live expressions, and a registered Boolean extension's inferred dtype and executed values. These eight shim cases pass within the stacked expression check: cargo +nightly-2026-04-16 test --locked -p vortex-datafusion --lib df_expression --no-default-features (16 cases total, 0.06s). Additional stacked tests check UDF-visible fields/scalar shape, lambda parameter bindings, and DF-owned CASE projection rewrites. Formatting uses CI-pinned nightly-2026-09-10.

Supersedes the membership-only #11426. @danking please review.

AI assistance: prepared with Codex and reviewed in the designated Codex review chat.

Stacked converter integration: #11432.

Signed-off-by: Daniel King <dan@spiraldb.com>
@codspeed

codspeed Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 3 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 1921 untouched benchmarks
⏩ 50 skipped benchmarks1


Comparing dk/df-expression-shim (6e97fda) with develop (d81bd4c)

Open in CodSpeed

Footnotes

  1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Signed-off-by: Daniel King <dan@spiraldb.com>
Signed-off-by: Daniel King <dan@spiraldb.com>
@danking danking added the changelog/feature A new feature label Nov 13, 2026
@danking
danking added this pull request to stack #11653 November 19, 2026 19:18
@danking

danking commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

@claude review

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @danking's task in 1m 20s —— View job


Review of #11431: typed DataFusion expression shim

  • Gather context (diff, surrounding code)
  • Review vortex-datafusion/src/df_expression.rs
  • Post review

The design is solid. It keeps decimal outputs in DF, rejects live, volatile and async expressions, checks the output type at runtime with vortex_ensure_eq!(result.data_type(), ...), and makes the options non-serializable. The tests compare against DF's own evaluation, including a real ArrayMap built by DF. I found one issue I'd fix before merging, plus a few smaller points.

1. Advisory mode swallows input execution errors, not just DF evaluation errors (worth fixing)

df_expression.rs:408-434. In advisory mode, any error from the closure becomes an all-true mask. The closure also contains:

  • array.execute_arrow(Some(field), ctx), which decodes the Vortex inputs and can surface IO, decompression or corruption errors.
  • Building the RecordBatch, including its nullability validation.
  • The result.len() and result.data_type() invariant checks.

So a corrupt segment or a type mismatch in the shim would silently disable the filter instead of failing the query. The PR description promises "retains rows on optional-filter evaluation failure". I'd narrow the fallback to that: convert the inputs and build the batch with ?, and apply the advisory fallback only to the evaluate(..).into_array_of_size(..) result. Keep the length and type checks fatal in both modes.

The ScalarFnVTable::is_infallible docs (vortex-array/src/scalar_fn/vtable.rs:319-320) also tell implementers to ignore incidental errors such as canonicalization failures. Turning them into true hides them, which is not the same thing.

Fix this →

There is also no direct test of advisory behavior in this PR. Nothing checks that a DF error becomes all-true, or that is_infallible() is true for advisory. If the stacked PR #11432 covers this, ignore this point. Otherwise a small raw_expression(.., advisory = true) test with the 1 / id lookup from preserves_key_evaluation_errors would cover it.

2. Ambiguous column names give inconsistent results: an error in one path, None in the other

df_expression.rs:102-113. A referenced column with a duplicated name returns plan_err!. A higher-order-function expression over a schema with any duplicated name returns Ok(None).

Physical schemas after joins can easily contain duplicate unqualified names such as id from both sides. With Err, the caller must handle failure explicitly instead of simply keeping the fragment in DF. Was the error intentional? If not, Ok(None) for ambiguity seems more consistent with the "decline when we can't represent it" contract. Keep Err for real invalid bindings, where the index and name disagree.

The second half of rejects_ambiguous_original_column_names (crossed_columns) tests an invalid binding, not ambiguity. Splitting it out or renaming the test would make the intent clearer.

3. rebind_columns wraps every non-leaf node and every Column in BoundField

df_expression.rs:162-181. Every rewritten node, including bare Columns, gets a BoundField wrapper. This should be correct, since with_new_children is forwarded and the CASE-local rebinding comment covers the tricky case. Two side effects are worth knowing:

  • Performance: DF exprs that pick fast paths by downcasting their children at construction will see BoundField instead of Column. For example, CaseExpr's eval-method selection runs again on with_new_children. The rebuilt tree is evaluated per batch, so this could cost something on hot projections.
  • Possible simplification: preserve_field only matters where the Display or return_field actually changes. A cheaper option is to wrap only the root, or only ScalarFunctionExpr nodes, whose UDF arg fields come from Display. If wrapping every node is required (for example because Column's Display includes @index), a short comment saying so would help the next reader.

4. Smaller points

  • Duplicate eligibility check: eligible() runs in both raw_expression and from_inputs. The second traversal also walks the BoundField-wrapped tree. This is harmless, but from_inputs is the shared entry point, so the check in raw_expression could go if Lower mixed DataFusion expressions into Vortex scans #11432 always goes through from_inputs.
  • Advisory fallback buffer: BooleanArray::from(vec![true; n]) (line 431) allocates a Vec<bool> first. BooleanArray::new(BooleanBuffer::new_set(n), None) avoids that.
  • Exact mode: is_infallible is false for every non-advisory expression, so infallible DF exprs such as IS NULL or the hash lookups won't get dictionary push-down. That is the safe default; it's just a note for later.
  • Pointer-based equality: Options and BoundField use pointer-based Eq and Hash on the PhysicalExpr. That's fine for process-local use, but two structurally identical wrappers won't deduplicate in CSE or caching. A one-line comment on why would help.
  • Test style: a few test functions have no blank line between them (lines 572-573, 606-607, 727-728, 790-791).

Checks

I only read the code. I ran no clippy, tests or formatting checks, in line with the repo's verification guidance. The PR description reports cargo test -p vortex-datafusion --lib df_expression --no-default-features passing; I did not re-run it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant