Repository navigation
Bind expressions before optimizing - #10002
Conversation
Merging this PR will degrade performance by 9.04%
|
db9209a to
c35e8ed
Compare
c35e8ed to
605db12
Compare
joseph-isaacs
left a comment
There was a problem hiding this comment.
i though we want simplify untyped too?
| let mut changed = false; | ||
|
|
||
| // Try simplify_untyped | ||
| if let Some(simplified) = expr.simplify_untyped_node()? { |
There was a problem hiding this comment.
so we don't have simplfiy untyped?
| } | ||
|
|
||
| /// Simplify the expression without type information. | ||
| pub(crate) fn simplify_untyped(&self, expr: &Expression) -> VortexResult<Option<Expression>> { |
|
@joseph-isaacs no functional difference now if we also have dtypes, but I can keep? |
605db12 to
5c985da
Compare
|
I think we want this for optimising an expr before binding even thought (vortex internals never use it). |
|
do we want to merge bind and optimize (optionally) |
|
@joseph-isaacs okay what we can do then is just keep the unbound expression optimizer but just never use it internally |
|
I think we should move strongly in this direction inside vortex core. And add a "vortex-expr" crate to handle unbound expressions for our "user friendly" APIs from Python or other language bindings. Host engines should never need to use the unbound API. And vortex-array also shouldn't know about it. This crate can handle coercion rules, literals, and so on as this implicit behavior can allow subtle bugs to leak in if engines use it. |
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
5c985da to
059e584
Compare
Replace separate eligibility and conversion checks with a schema-aware `try_convert`, preserve SQL null semantics for IN/NOT IN, and reject pushdown where casts, short-circuit evaluation, or nullable struct access could differ. Re-split filters after per-file schema adaptation so unsupported conjuncts run as DataFusion residual filters before projection and limit. Fix v2 filter pushdown after projection swapping, and rename `ExpressionConvertor` to `ExpressionConverter`. Squashed onto develop, porting the bind-before-optimize change from #10002. Signed-off-by: Adam Gutglick <adam@spiraldb.com> Signed-off-by: "Claude" <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013wUbDvL9BL68K3YceQiuZw
Develop changed `opener.rs` after this branch removed it. Apply the same changes to the morsel planner: - Let `FilePruner` decide if pruning is useful (#9841). - Share the physical Arrow schema of each file across partitions. - Disable projection pushdown for projections with lambdas (#10215). - Bind expressions before optimizing them (#10002). - Explain why the limit applies after the filter (#8766). - Use `ScanBuilder<ArrayRef>` and `RepeatedScan<ArrayRef>`, because develop keeps `ScanBuilder::map`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
Replace separate eligibility and conversion checks with a schema-aware `try_convert`, preserve SQL null semantics for IN/NOT IN, and reject pushdown where casts, short-circuit evaluation, or nullable struct access could differ. Re-split filters after per-file schema adaptation so unsupported conjuncts run as DataFusion residual filters before projection and limit. Fix v2 filter pushdown after projection swapping, and rename `ExpressionConvertor` to `ExpressionConverter`. Squashed onto develop, porting the bind-before-optimize change from #10002. Signed-off-by: Adam Gutglick <adam@spiraldb.com> Signed-off-by: "Claude" <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013wUbDvL9BL68K3YceQiuZw Signed-off-by: Robert Kruszewski <github@robertk.io>
Supersedes #9514.
Optimization now runs only on
BoundExpressions: callers bind first, then optimize. This removes the optimizer-owned dtype cache, rules can read dtypes straight off the tree, and it unblocks lambdas/variables, which must be typed before optimization.Changes
BoundExpression::optimizeandoptimize_recursivereplace theExpressionversions. Traversal and the fixpoint loop are unchanged.ScalarFnVTable::simplifynow takes a&BoundExpression, andsimplify_untypedis merged into it.SimplifyCtxandSimplifyCacheare removed.ExpressionReduceNodekeeps its name andreduce_expressiondispatch, but now borrows a bound tree.find_between.GetItem::simplify_untypedis removed because it duplicatedGetItem::reduce. If the rebuiltBetweenfails to type-check,find_betweennow skips the rewrite instead of producing a tree that fails to bind.expr.optimize_recursive(dtype)?.bind(dtype)toexpr.bind(dtype)?.optimize_recursive().expr::transform::partitionis deleted. It was the last user of the unbound optimizer and has no callers; layouts usebound_partition. Remove unused unbound expression analysis and traversal helpers #9993 deletes it too, so whichever PR lands second will have a trivial conflict.API changes
Expression::optimize(scope)/optimize_recursive(scope)becomeBoundExpression::optimize()/optimize_recursive().ScalarFnVTable::simplify(options, &BoundExpression)replacessimplify(options, &Expression, &dyn SimplifyCtx)andsimplify_untyped.ExpressionReduceNode::new(&BoundExpression)drops the scope argument.partition,partition_annotationsandPartitionedExprare removed.Behavior
I checked this with a differential corpus of 30,018 generated expressions targeting every rule, comparing develop's
optimize + bindagainst the newbind + optimizeand evaluating before/after on data:Masksimplify calledvortex_expecton a null mask literal) and now fail cleanly at bind time.SimplifyCachekeys dtypes byExpression, andScalarequality ignores nullability, so develop reused the wrong cached dtype and droppedfill_nulls it should have kept.No well-typed expression regressed.
Existing bugs found, not addressed here:
CASE WHEN→fill_nullrewrite is unexecutable for utf8.select([], nullable_struct)drops the struct's validity.CaseWhenexecution can return a result that is non-nullable when its dtype is nullable.and/or/zipliteral rules can change the result's nullability.Performance
The optimizer benchmarks bind outside the timed loop and measure only
optimize_recursive, the same work develop's benchmarks time. Local medians:or_chain200balanced_or64–256 idsbalanced_or1 idin_listSmall predicates pay a constant ~0.15–0.2 µs in the optimizer, almost all of it in
find_between, which now type-checks theand/Betweennodes it rebuilds (0.56 µs vs 0.34 µs). Across the fullbind + optimizepipeline that callers run, the gap is ~0.3–0.45 µs, because binding also types the comparisonsfind_betweenlater replaces. Closing it would need an uncheckedBoundExpressionconstructor. Open question.Testing
cargo nextest run -p vortex-array -p vortex-layout -p vortex-file -p vortex -p vortex-datafusion -p vortex-duckdbcargo test --doc -p vortex -p vortex-arraycargo clippy --all-targets --all-features -- -D warningson all changed crates andfuzz/file_iocargo check --workspace --all-targetsNot run: Python tests.
vortex-cuda-ffiwas not type-checked because it is compiled out on macOS.