Skip to content

Fix 3635 redundant projections - #3861

Closed
alex-spies wants to merge 3 commits into
apache:masterfrom
alex-spies:fix_3635_redundant_projections
Closed

alex-spies wants to merge 3 commits into
apache:masterfrom
alex-spies:fix_3635_redundant_projections

Conversation

@alex-spies

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #3635.

Rationale for this change

If CommonSubexprEliminate identifies a common sub-expression used in a Filter node, it will add a projection before a filter which adds a new column with the common sub-expression. If the FilterPushdown is applied afterwards, the filter can be pushed past the additional projection - now the additional projection is completely redundant and, instead of optimizing performance, actually worsens it.

What changes are included in this PR?

  • Move CommonSubexprEliminate after FilterPushdown in the default optimization rules.

Are there any user-facing changes?

No API changes, but optimizing a LogicalPlan will in some cases produce a different (hopefully better) result.

@alex-spies

Copy link
Copy Markdown
Contributor Author

Only a draft for now; we need to take a look whether the new expected LogicalPlans in the tests make 100% sense.

@alamb , what do you think?

@@ -1,9 +1,8 @@
Projection: SUM(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount) AS revenue
Aggregate: groupBy=[[]], aggr=[[SUM(CAST(lineitem.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))) AS SUM(lineitem.l_extendedprice * Int64(1) - lineitem.l_discount)]]
Projection: lineitem.l_extendedprice, lineitem.l_discount

@Dandandan Dandandan Oct 17, 2022 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not really redundant I think (when converting to a physical aggregation, we don't want to group / hash repartition the other columns)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I don't think the hash aggregation takes columns into account which are neither in the groupBy nor in the aggr expressions. Even if it did, it probably shouldn't be the job of CommonSubexprEliminate to ensure that redundant columns are projected out first, that should be it's own (physical?) optimization rule IMO.

@@ -1,6 +1,5 @@
Projection: SUM(lineitem.l_extendedprice * lineitem.l_discount) AS revenue
Aggregate: groupBy=[[]], aggr=[[SUM(lineitem.l_extendedprice * lineitem.l_discount)]]
Projection: lineitem.l_extendedprice, lineitem.l_discount

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

@alex-spies

Copy link
Copy Markdown
Contributor Author

@Dandandan sorry, I could have been more specific in the PR description.

The issue #3635 is mostly about one test that is currently ignored, namely multiple_or_predicates.

On current master (feff5dc) the produced logical plan is the following:

[
    "Explain [plan_type:Utf8, plan:Utf8]",
    "  Projection: lineitem.l_partkey [l_partkey:Int64]",
    "    Projection: lineitem.l_partkey [l_partkey:Int64]",
    "      Filter: part.p_brand = Utf8(\"Brand#12\") AND lineitem.l_quantity >= Decimal128(Some(100),15,2) AND lineitem.l_quantity <= Decimal128(Some(1100),15,2) AND part.p_size <= Int32(5) OR part.p_brand = Utf8(\"Brand#23\") AND lineitem.l_quantity >= Decimal128(Some(1000),15,2) AND lineitem.l_quantity <= Decimal128(Some(2000),15,2) AND part.p_size <= Int32(10) OR part.p_brand = Utf8(\"Brand#34\") AND lineitem.l_quantity >= Decimal128(Some(2000),15,2) AND lineitem.l_quantity <= Decimal128(Some(3000),15,2) AND part.p_size <= Int32(15) [l_partkey:Int64, l_quantity:Decimal128(15, 2), p_partkey:Int64, p_brand:Utf8, p_size:Int32]",
    "        Inner Join: lineitem.l_partkey = part.p_partkey [l_partkey:Int64, l_quantity:Decimal128(15, 2), p_partkey:Int64, p_brand:Utf8, p_size:Int32]",
    "          Filter: lineitem.l_quantity >= Decimal128(Some(100),15,2) AND lineitem.l_quantity <= Decimal128(Some(1100),15,2) OR lineitem.l_quantity >= Decimal128(Some(1000),15,2) AND lineitem.l_quantity <= Decimal128(Some(2000),15,2) OR lineitem.l_quantity >= Decimal128(Some(2000),15,2) AND lineitem.l_quantity <= Decimal128(Some(3000),15,2) [l_partkey:Int64, l_quantity:Decimal128(15, 2)]",
    "            TableScan: lineitem projection=[l_partkey, l_quantity], partial_filters=[lineitem.l_quantity >= Decimal128(Some(100),15,2) AND lineitem.l_quantity <= Decimal128(Some(1100),15,2) OR lineitem.l_quantity >= Decimal128(Some(1000),15,2) AND lineitem.l_quantity <= Decimal128(Some(2000),15,2) OR lineitem.l_quantity >= Decimal128(Some(2000),15,2) AND lineitem.l_quantity <= Decimal128(Some(3000),15,2)] [l_partkey:Int64, l_quantity:Decimal128(15, 2)]",
    "          Filter: part.p_size >= Int32(1) AND part.p_brand = Utf8(\"Brand#12\") AND part.p_size <= Int32(5) OR part.p_brand = Utf8(\"Brand#23\") AND part.p_size <= Int32(10) OR part.p_brand = Utf8(\"Brand#34\") AND part.p_size <= Int32(15) [p_partkey:Int64, p_brand:Utf8, p_size:Int32]",
    "            TableScan: part projection=[p_partkey, p_brand, p_size], partial_filters=[part.p_size >= Int32(1), part.p_brand = Utf8(\"Brand#12\") AND part.p_size <= Int32(5) OR part.p_brand = Utf8(\"Brand#23\") AND part.p_size <= Int32(10) OR part.p_brand = Utf8(\"Brand#34\") AND part.p_size <= Int32(15)] [p_partkey:Int64, p_brand:Utf8, p_size:Int32]",
]

There is a double projection at the top of the plan; one of the projection nodes was introduced by CommonSubexprEliminate.

The actual logical used to be even more nonsensical when #3635 was opened, c.f. our discussion here.

Comment on lines +17 to +18
Filter: nation.n_name = Utf8("SAUDI ARABIA")Utf8("SAUDI ARABIA")nation.n_name
Projection: nation.n_name = Utf8("SAUDI ARABIA") AS nation.n_name = Utf8("SAUDI ARABIA")Utf8("SAUDI ARABIA")nation.n_name, nation.n_nationkey

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit surprising that apparently CommonSubexprEliminate now introduces this projection here even though the expression is used only in one filter.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps @waynexia has some insight

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I try EXPLAIN VERBOSE and it looks like the FilterPushdown rule generates a redundant expr. Here is the log:

| logical_plan after reduce_cross_join                       | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                  |
| logical_plan after common_sub_expression_eliminate         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                  |
| logical_plan after eliminate_limit                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                  |
| logical_plan after projection_push_down                    | Sort: numwait DESC NULLS FIRST, supplier.s_name ASC NULLS LAST                                                                                                                                                                                                                      |
|                                                            |   Projection: supplier.s_name, COUNT(UInt8(1)) AS numwait                                                                                                                                                                                                                           |
|                                                            |     Aggregate: groupBy=[[supplier.s_name]], aggr=[[COUNT(UInt8(1))]]                                                                                                                                                                                                                |
|                                                            |       Filter: orders.o_orderstatus = Utf8("F") AND l1.l_receiptdate > l1.l_commitdate AND nation.n_name = Utf8("SAUDI ARABIA")                                                                                                                                                      |
|                                                            |         Anti Join: l1.l_orderkey = l3.l_orderkey Filter: l3.l_suppkey != l1.l_suppkey                                                                                                                                                                                               |
|                                                            |           Filter: orders.o_orderstatus = Utf8("F") AND l1.l_receiptdate > l1.l_commitdate AND nation.n_name = Utf8("SAUDI ARABIA")                                                                                                                                                  |
|                                                            |             Semi Join: l1.l_orderkey = l2.l_orderkey Filter: l2.l_suppkey != l1.l_suppkey                                                                                                                                                                                           |
|                                                            |               Inner Join: supplier.s_nationkey = nation.n_nationkey                                                                                                                                                                                                                 |
|                                                            |                 Inner Join: l1.l_orderkey = orders.o_orderkey                                                                                                                                                                                                                       |
|                                                            |                   Inner Join: supplier.s_suppkey = l1.l_suppkey                                                                                                                                                                                                                     |
|                                                            |                     TableScan: supplier projection=[s_suppkey, s_name, s_nationkey]                                                                                                                                                                                                 |
|                                                            |                     SubqueryAlias: l1                                                                                                                                                                                                                                               |
|                                                            |                       TableScan: lineitem projection=[l_orderkey, l_suppkey, l_commitdate, l_receiptdate]                                                                                                                                                                           |
|                                                            |                   TableScan: orders projection=[o_orderkey, o_orderstatus]                                                                                                                                                                                                          |
|                                                            |                 TableScan: nation projection=[n_nationkey, n_name]                                                                                                                                                                                                                  |
|                                                            |               SubqueryAlias: l2                                                                                                                                                                                                                                                     |
|                                                            |                 TableScan: lineitem projection=[l_orderkey, l_suppkey]                                                                                                                                                                                                              |
|                                                            |           Filter: l3.l_receiptdate > l3.l_commitdate                                                                                                                                                                                                                                |
|                                                            |             SubqueryAlias: l3                                                                                                                                                                                                                                                       |
|                                                            |               TableScan: lineitem projection=[l_orderkey, l_suppkey, l_commitdate, l_receiptdate]                                                                                                                                                                                   |
| logical_plan after rewrite_disjunctive_predicate           | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                  |
| logical_plan after reduce_outer_join                       | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                  |
| logical_plan after filter_push_down                        | Sort: numwait DESC NULLS FIRST, supplier.s_name ASC NULLS LAST                                                                                                                                                                                                                      |
|                                                            |   Projection: supplier.s_name, COUNT(UInt8(1)) AS numwait                                                                                                                                                                                                                           |
|                                                            |     Aggregate: groupBy=[[supplier.s_name]], aggr=[[COUNT(UInt8(1))]]                                                                                                                                                                                                                |
|                                                            |       Anti Join: l1.l_orderkey = l3.l_orderkey Filter: l3.l_suppkey != l1.l_suppkey                                                                                                                                                                                                 |
|                                                            |         Semi Join: l1.l_orderkey = l2.l_orderkey Filter: l2.l_suppkey != l1.l_suppkey                                                                                                                                                                                               |
|                                                            |           Inner Join: supplier.s_nationkey = nation.n_nationkey                                                                                                                                                                                                                     |
|                                                            |             Inner Join: l1.l_orderkey = orders.o_orderkey                                                                                                                                                                                                                           |
|                                                            |               Inner Join: supplier.s_suppkey = l1.l_suppkey                                                                                                                                                                                                                         |
|                                                            |                 TableScan: supplier projection=[s_suppkey, s_name, s_nationkey]                                                                                                                                                                                                     |
|                                                            |                 Filter: l1.l_receiptdate > l1.l_commitdate AND l1.l_receiptdate > l1.l_commitdate                                                                                                                                                                                   |
|                                                            |                   SubqueryAlias: l1                                                                                                                                                                                                                                                 |
|                                                            |                     TableScan: lineitem projection=[l_orderkey, l_suppkey, l_commitdate, l_receiptdate]                                                                                                                                                                             |
|                                                            |               Filter: orders.o_orderstatus = Utf8("F") AND orders.o_orderstatus = Utf8("F")                                                                                                                                                                                         |
|                                                            |                 TableScan: orders projection=[o_orderkey, o_orderstatus]                                                                                                                                                                                                            |
|                                                            |             Filter: nation.n_name = Utf8("SAUDI ARABIA") AND nation.n_name = Utf8("SAUDI ARABIA")                                                                                                                                                                                   |
|                                                            |               TableScan: nation projection=[n_nationkey, n_name]                                                                                                                                                                                                                    |
|                                                            |           SubqueryAlias: l2                                                                                                                                                                                                                                                         |
|                                                            |             TableScan: lineitem projection=[l_orderkey, l_suppkey]                                                                                                                                                                                                                  |
|                                                            |         Filter: l3.l_receiptdate > l3.l_commitdate                                                                                                                                                                                                                                  |
|                                                            |           SubqueryAlias: l3                                                                                                                                                                                                                                                         |
|                                                            |             TableScan: lineitem projection=[l_orderkey, l_suppkey, l_commitdate, l_receiptdate]                                                                                                                                                                                     |
| logical_plan after limit_push_down                         | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                  |
| logical_plan after single_distinct_aggregation_to_group_by | SAME TEXT AS ABOVE                                                                                                                                                                                                                                                                  |
| logical_plan after type_coercion                           | SAME TEXT AS ABOVE             

And the redundant AND in filter is eliminated in the following simplify_expressions rule, but the projection introduced by CommonSubexpressionEliminate left.

I haven't dug into it. But this is not in the scope of this PR I suppose. I'll open an issue to track it. (the causality among rules are really interesting😋)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened #3914

@alex-spies

Copy link
Copy Markdown
Contributor Author

I'm closing this as #3635 has been closed by some other means, so there's probably no point in messing with the optimization rule order then.

@alex-spies alex-spies closed this Dec 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

after type coercion CommonSubexprEliminate will produce invalid projection

4 participants