You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The (logical) optimizer contains some support for folding (boolean) constants. This can help, especially with other optimization passes, to optimize queries. For example, LIMIT (0 + 0) could be optimized first to LIMIT 0, to enable eliminating the whole plan.
We should try to extend this support to most datatypes & expressions.
Describe the solution you'd like Exprs can already be evaluated against a RecordBatch, and there is code to evaluate scalar values without going through Arrow. To make sure that the constant evaluation is implemented correctly & the same as the evaluation code, we should be able to reuse the code from there.
Describe alternatives you've considered
Manually implement the constant folding support. Downside here is that we end up with two implementations, which has a higher maintenance burden.
Additional context
Not in scope: add it to physical optimizer too. Here it could help too, especially if we have support for partitions.
@alamb just picking your brain here - do you think this should be part of the logical optimizations or physical optimizations?
I think the suggested route (re-use evaluation code) is only feasible for the physical optimization, not the logical optimization rules.
A way that could work within the current setup for PhysicalExpr:
Evaluation is implemented for PhysicalExprs not Exprs. An empty RecordBatch could be used to call into the code and extract the scalar values (if any).
This makes it a bit less useful (still useful nonetheless), as some other optimizations might benefit from constant folding
I am wondering here in general, whether we can/should unify LogicalPlan/PhysicalPlanExpr/PhysicalExpr a bit more in order to not have to write two versions of the same thing or being limited in the optimizations / optimization order.
fwiw, when a logical optimization is applied, the expressions are re-written and the "column name" is consequently re-written. Thus, what was named LIMIT (0 + 0) becomes LIMIT 0.
To apply it on the logical level, we may need to wrap the expression by an .alias for it to preserve the column name.
I agree that the sooner in the optimization these are applied, the higher the likelihood of synergies between optimizers.
@alamb just picking your brain here - do you think this should be part of the logical optimizations or physical optimizations?
I think the suggested route (re-use evaluation code) is only feasible for the physical optimization, not the logical optimization rules.
I would imagine this to be done on Exprs, not PhysicalExprs to allow the rewritten expressions to be used as much as possible by other optimization passes (e.g. filter and projection pushdown, which is done at the LogicalPlan level)
I am wondering here in general, whether we can/should unify LogicalPlan/PhysicalPlan Expr/PhysicalExpr a bit more in order to not have to write two versions of the same thing or being limited in the optimizations / optimization order.
I think the LogicalPlan / PhysicalPlan distinction makes sense (b/c logically a Join is just a Join -- but physically maybe we would be using a CROSS JOIN w/ filter, or an Hash Inner Join, or a Merge Join, etc)
I am not as sure about the distinction between Expr and PhysicalExpr -- I haven't looked carefully at the code to know what additional information a PhysicalExpr needs that an Expr doesn't have -- and you can make a PhysicalExpr from an Expr and a Schemacode link
If we could directly evaluate Exprs without having to apply a transformation to them that would be pretty cool (and clean up a lot of duplication I think)
Activity
alamb commentedon May 3, 2021
This would be a neat feature.
Dandandan commentedon May 6, 2021
@alamb just picking your brain here - do you think this should be part of the logical optimizations or physical optimizations?
I think the suggested route (re-use evaluation code) is only feasible for the physical optimization, not the logical optimization rules.
A way that could work within the current setup for
PhysicalExpr:PhysicalExprs notExprs. An emptyRecordBatchcould be used to call into the code and extract the scalar values (if any).This makes it a bit less useful (still useful nonetheless), as some other optimizations might benefit from constant folding
I am wondering here in general, whether we can/should unify
LogicalPlan/PhysicalPlanExpr/PhysicalExpra bit more in order to not have to write two versions of the same thing or being limited in the optimizations / optimization order.jorgecarleitao commentedon May 6, 2021
fwiw, when a logical optimization is applied, the expressions are re-written and the "column name" is consequently re-written. Thus, what was named
LIMIT (0 + 0)becomesLIMIT 0.To apply it on the logical level, we may need to wrap the expression by an
.aliasfor it to preserve the column name.I agree that the sooner in the optimization these are applied, the higher the likelihood of synergies between optimizers.
Dandandan commentedon May 6, 2021
@jorgecarleitao that's a good one - I did also see something in the same order recently when looking at this #268 problem.
Dandandan commentedon May 6, 2021
It is a problem already in the current constant folding! I am opening an issue for this.
alamb commentedon May 7, 2021
@Dandandan
I would imagine this to be done on
Exprs, notPhysicalExprsto allow the rewritten expressions to be used as much as possible by other optimization passes (e.g. filter and projection pushdown, which is done at theLogicalPlanlevel)I think the LogicalPlan / PhysicalPlan distinction makes sense (b/c logically a Join is just a Join -- but physically maybe we would be using a CROSS JOIN w/ filter, or an Hash Inner Join, or a Merge Join, etc)
I am not as sure about the distinction between
ExprandPhysicalExpr-- I haven't looked carefully at the code to know what additional information aPhysicalExprneeds that anExprdoesn't have -- and you can make aPhysicalExprfrom anExprand aSchemacode linkIf we could directly evaluate
Exprswithout having to apply a transformation to them that would be pretty cool (and clean up a lot of duplication I think)alamb commentedon May 19, 2021
FYI while I was reviewing the code in https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/physical_plan/parquet.rs in the context of #363 I noticed there is already a way to do "partial evaluation" for expressions -- maybe we could fake the same to evaluate
Exprsthat have no inputs -- pass in a 1 row null array as input or something.alamb commentedon Oct 11, 2021
Possibly related to #1070
alamb commentedon Sep 12, 2022
I think we have implemented most of the suggestions in this issue -- I am not sure if it is tracking anything actionable anymore
Dandandan commentedon Sep 14, 2022
Yes I agree, this is done 🚀. Closing this issue
doc: Update supported expressions (apache#237)