Repository navigation
Fix 3635 redundant projections - #3861
alex-spies wants to merge 3 commits into
Conversation
|
Only a draft for now; we need to take a look whether the new expected @alamb , what do you think? |
| @@ -1,9 +1,8 @@ | |||
| Projection: SUM(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount) AS revenue | |||
| Aggregate: groupBy=[[]], aggr=[[SUM(CAST(lineitem.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))) AS SUM(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)]] | |||
| Projection: lineitem.l_extendedprice, lineitem.l_discount | |||
There was a problem hiding this comment.
This is not really redundant I think (when converting to a physical aggregation, we don't want to group / hash repartition the other columns)?
There was a problem hiding this comment.
Hm, I don't think the hash aggregation takes columns into account which are neither in the groupBy nor in the aggr expressions. Even if it did, it probably shouldn't be the job of CommonSubexprEliminate to ensure that redundant columns are projected out first, that should be it's own (physical?) optimization rule IMO.
| @@ -1,6 +1,5 @@ | |||
| Projection: SUM(lineitem.l_extendedprice * lineitem.l_discount) AS revenue | |||
| Aggregate: groupBy=[[]], aggr=[[SUM(lineitem.l_extendedprice * lineitem.l_discount)]] | |||
| Projection: lineitem.l_extendedprice, lineitem.l_discount | |||
|
@Dandandan sorry, I could have been more specific in the PR description. The issue #3635 is mostly about one test that is currently ignored, namely On current master (feff5dc) the produced logical plan is the following: There is a double projection at the top of the plan; one of the projection nodes was introduced by The actual logical used to be even more nonsensical when #3635 was opened, c.f. our discussion here. |
| Filter: nation.n_name = Utf8("SAUDI ARABIA")Utf8("SAUDI ARABIA")nation.n_name | ||
| Projection: nation.n_name = Utf8("SAUDI ARABIA") AS nation.n_name = Utf8("SAUDI ARABIA")Utf8("SAUDI ARABIA")nation.n_name, nation.n_nationkey |
There was a problem hiding this comment.
It's a bit surprising that apparently CommonSubexprEliminate now introduces this projection here even though the expression is used only in one filter.
There was a problem hiding this comment.
I try EXPLAIN VERBOSE and it looks like the FilterPushdown rule generates a redundant expr. Here is the log:
| logical_plan after reduce_cross_join | SAME TEXT AS ABOVE |
| logical_plan after common_sub_expression_eliminate | SAME TEXT AS ABOVE |
| logical_plan after eliminate_limit | SAME TEXT AS ABOVE |
| logical_plan after projection_push_down | Sort: numwait DESC NULLS FIRST, supplier.s_name ASC NULLS LAST |
| | Projection: supplier.s_name, COUNT(UInt8(1)) AS numwait |
| | Aggregate: groupBy=[[supplier.s_name]], aggr=[[COUNT(UInt8(1))]] |
| | Filter: orders.o_orderstatus = Utf8("F") AND l1.l_receiptdate > l1.l_commitdate AND nation.n_name = Utf8("SAUDI ARABIA") |
| | Anti Join: l1.l_orderkey = l3.l_orderkey Filter: l3.l_suppkey != l1.l_suppkey |
| | Filter: orders.o_orderstatus = Utf8("F") AND l1.l_receiptdate > l1.l_commitdate AND nation.n_name = Utf8("SAUDI ARABIA") |
| | Semi Join: l1.l_orderkey = l2.l_orderkey Filter: l2.l_suppkey != l1.l_suppkey |
| | Inner Join: supplier.s_nationkey = nation.n_nationkey |
| | Inner Join: l1.l_orderkey = orders.o_orderkey |
| | Inner Join: supplier.s_suppkey = l1.l_suppkey |
| | TableScan: supplier projection=[s_suppkey, s_name, s_nationkey] |
| | SubqueryAlias: l1 |
| | TableScan: lineitem projection=[l_orderkey, l_suppkey, l_commitdate, l_receiptdate] |
| | TableScan: orders projection=[o_orderkey, o_orderstatus] |
| | TableScan: nation projection=[n_nationkey, n_name] |
| | SubqueryAlias: l2 |
| | TableScan: lineitem projection=[l_orderkey, l_suppkey] |
| | Filter: l3.l_receiptdate > l3.l_commitdate |
| | SubqueryAlias: l3 |
| | TableScan: lineitem projection=[l_orderkey, l_suppkey, l_commitdate, l_receiptdate] |
| logical_plan after rewrite_disjunctive_predicate | SAME TEXT AS ABOVE |
| logical_plan after reduce_outer_join | SAME TEXT AS ABOVE |
| logical_plan after filter_push_down | Sort: numwait DESC NULLS FIRST, supplier.s_name ASC NULLS LAST |
| | Projection: supplier.s_name, COUNT(UInt8(1)) AS numwait |
| | Aggregate: groupBy=[[supplier.s_name]], aggr=[[COUNT(UInt8(1))]] |
| | Anti Join: l1.l_orderkey = l3.l_orderkey Filter: l3.l_suppkey != l1.l_suppkey |
| | Semi Join: l1.l_orderkey = l2.l_orderkey Filter: l2.l_suppkey != l1.l_suppkey |
| | Inner Join: supplier.s_nationkey = nation.n_nationkey |
| | Inner Join: l1.l_orderkey = orders.o_orderkey |
| | Inner Join: supplier.s_suppkey = l1.l_suppkey |
| | TableScan: supplier projection=[s_suppkey, s_name, s_nationkey] |
| | Filter: l1.l_receiptdate > l1.l_commitdate AND l1.l_receiptdate > l1.l_commitdate |
| | SubqueryAlias: l1 |
| | TableScan: lineitem projection=[l_orderkey, l_suppkey, l_commitdate, l_receiptdate] |
| | Filter: orders.o_orderstatus = Utf8("F") AND orders.o_orderstatus = Utf8("F") |
| | TableScan: orders projection=[o_orderkey, o_orderstatus] |
| | Filter: nation.n_name = Utf8("SAUDI ARABIA") AND nation.n_name = Utf8("SAUDI ARABIA") |
| | TableScan: nation projection=[n_nationkey, n_name] |
| | SubqueryAlias: l2 |
| | TableScan: lineitem projection=[l_orderkey, l_suppkey] |
| | Filter: l3.l_receiptdate > l3.l_commitdate |
| | SubqueryAlias: l3 |
| | TableScan: lineitem projection=[l_orderkey, l_suppkey, l_commitdate, l_receiptdate] |
| logical_plan after limit_push_down | SAME TEXT AS ABOVE |
| logical_plan after single_distinct_aggregation_to_group_by | SAME TEXT AS ABOVE |
| logical_plan after type_coercion | SAME TEXT AS ABOVE
And the redundant AND in filter is eliminated in the following simplify_expressions rule, but the projection introduced by CommonSubexpressionEliminate left.
I haven't dug into it. But this is not in the scope of this PR I suppose. I'll open an issue to track it. (the causality among rules are really interesting😋)
|
I'm closing this as #3635 has been closed by some other means, so there's probably no point in messing with the optimization rule order then. |
Which issue does this PR close?
Closes #3635.
Rationale for this change
If
CommonSubexprEliminateidentifies a common sub-expression used in aFilternode, it will add a projection before a filter which adds a new column with the common sub-expression. If theFilterPushdownis applied afterwards, the filter can be pushed past the additional projection - now the additional projection is completely redundant and, instead of optimizing performance, actually worsens it.What changes are included in this PR?
CommonSubexprEliminateafterFilterPushdownin the default optimization rules.Are there any user-facing changes?
No API changes, but optimizing a
LogicalPlanwill in some cases produce a different (hopefully better) result.