Skip to content

improve Filter pushdown to Join - #5770

Merged
Dandandan merged 1 commit into
apache:mainfrom
mingmwang:tpch17
Apr 1, 2023
Merged

Dandandan merged 1 commit into
apache:mainfrom
mingmwang:tpch17

Conversation

@mingmwang

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

Improve some TPCH query performance, simply the generate logical plan and physical plan.

What changes are included in this PR?

  1. Convert filters to join filters for Inner Join
  2. Avoid duplicated filters
  3. Fixed unstable physical HashJoin plan

tpch-q7, tpch-q17, tpch-q19, tpch-q20 are impacted by this PR.

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions Bot added core Core DataFusion crate optimizer Optimizer rules labels Mar 29, 2023
@mingmwang

Copy link
Copy Markdown
Contributor Author

@yahoNanJing
Please help me to review

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the plan changes and the code carefully -- nice work @mingmwang

| logical_plan | Projection: CAST(SUM(lineitem.l_extendedprice) AS Float64) / Float64(7) AS avg_yearly |
| | Aggregate: groupBy=[[]], aggr=[[SUM(lineitem.l_extendedprice)]] |
| | Projection: lineitem.l_extendedprice |
| | Inner Join: part.p_partkey = __scalar_sq_1.l_partkey Filter: CAST(lineitem.l_quantity AS Decimal128(30, 15)) < CAST(__scalar_sq_1.__value AS Decimal128(30, 15)) |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a better plan because the redundant

Filter: part.p_partkey = lineitem.l_partkey AND lineitem.l_partkey = part.p_partkey

, which is already done by the earlier joins, is removed, right?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It also pushes the filter

|               |       Filter: CAST(lineitem.l_quantity AS Decimal128(30, 15)) < CAST(__scalar_sq_1.__value AS Decimal128(30, 15)) AND __scalar_sq_1.l_partkey = lineitem.l_partkey              |
``

Into the Join which seems like a win to me (avoid generating output)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a better plan because the redundant

Filter: part.p_partkey = lineitem.l_partkey AND lineitem.l_partkey = part.p_partkey

, which is already done by the earlier joins, is removed, right?

Yes, the duplicated filters are removed. Actually why the original plan include duplicate filters is because the push_down_filter rule infers additional filters and try to pushdown them down. If they can not be pushed down, those inferred filters are added back to the Filters, this is unnecessary, need to differ the inferred filters and the original filters.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for the explanation.

let cols = expr.to_columns()?;

// Collect left & right field indices
// Collect left & right field indices, the field indices are sorted in ascending order

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the sort order is important for later stages, can you make a note about the rationale (so the comment explains why the sorting is important, in addition to noting the output is sorted)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will do.

@andygrove

Copy link
Copy Markdown
Member

@mingmwang Could you share any performance numbers for the improvements for the affected queries?

@mingmwang

Copy link
Copy Markdown
Contributor Author

@mingmwang Could you share any performance numbers for the improvements for the affected queries?

Sure, will do. unfortunately, the performance improvement is just a little. For q17, the major bottleneck is still the Aggregation.

@jackwener jackwener left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job to me. Thanks @mingmwang .

@jackwener

jackwener commented Mar 31, 2023 •

Copy link
Copy Markdown
Member

This PR remind me. I also notice some optimization.

We can do a EPIC list tasks about optimizer to collect those optimization like #5546.

such as predicate move around ......

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants