Skip to content

Fix required partitioning of Single aggregation mode - #6950

Merged
Dandandan merged 9 commits into
mainfrom
fix_single_required_partitioning
Jul 15, 2023
Merged

Dandandan merged 9 commits into
mainfrom
fix_single_required_partitioning

Conversation

@Dandandan

@Dandandan Dandandan commented Jul 13, 2023 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6949

Rationale for this change

Fix a small issue in this aggregation mode .
Currently the Single mode returns UnspecifiedDistribution as required distribution, which is not correct. If we would re-optimize / change the plan, it could cause the plan to return incorrect results. This currently is not the case in DataFusion as we rely on the fact that required distribution is already satisfied and we don't modify that part of the plan.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions Bot added the core Core DataFusion crate label Jul 13, 2023
@Dandandan

Copy link
Copy Markdown
Contributor Author

FYI @mingmwang

@mustafasrepo

Copy link
Copy Markdown
Contributor

According to CombinePartialFinalAggregate rule, when planner sees AggregateMode::Partial + AggregateMode::Final or AggregateMode::Partial + AggregateMode::FinalPartitioned, it replaces these immediate operators with AggregateMode::Single.
For second case changing distribution requirement to Distribution::HashPartitioned makes sense. However, for first case distribution requirement should be SinglePartition I guess.

With the proposed changes, AggregateMode::Single will require vec![Distribution::HashPartitioned(vec![])] (where group by is empty). when it is constructed from AggregateMode::Partial + AggregateMode::Final stack.

Will this requirement add RepartitionExec(hash) during planning?

@mingmwang

Copy link
Copy Markdown
Contributor

@mustafasrepo

I think the purpose of this PR is to extend the use case of AggregateMode::Single. I think the change make sense, but it is better to add some comments.

@mingmwang mingmwang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Dandandan

Copy link
Copy Markdown
Contributor Author

According to CombinePartialFinalAggregate rule, when planner sees AggregateMode::Partial + AggregateMode::Final or AggregateMode::Partial + AggregateMode::FinalPartitioned, it replaces these immediate operators with AggregateMode::Single. For second case changing distribution requirement to Distribution::HashPartitioned makes sense. However, for first case distribution requirement should be SinglePartition I guess.

With the proposed changes, AggregateMode::Single will require vec![Distribution::HashPartitioned(vec![])] (where group by is empty). when it is constructed from AggregateMode::Partial + AggregateMode::Final stack.

Will this requirement add RepartitionExec(hash) during planning?

Ah I missed that part.
I missed the part that Final + FinalPartitioned is also being converted to Single. I think the correct way (that avoids planning errors) would be a Single that requires SinglePartition in that case.

Currently the optimization rule runs after the RepartitionExec(hash) and CoalescePartitions are added so it should already have those requirements, but I think it's better to generate a correct plan, otherwise we could introduce errors later on (e.g. by a rule that removes unnecessary RepartitionExec / CoalescePartitions or by using the Single mode directly before the required partitioning is added).

@Dandandan

Copy link
Copy Markdown
Contributor Author

I added a mode SinglePartitioned which requires the input to be partitioned, the existing Single mode does require the input to be coalesced into one partition.

@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Jul 13, 2023
Daniël Heres added 2 commits July 13, 2023 23:44

@mustafasrepo mustafasrepo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Dandandan for this PR.

@Dandandan
Dandandan merged commit 9289c20 into main Jul 15, 2023
@Dandandan
Dandandan deleted the fix_single_required_partitioning branch August 12, 2023 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Single aggregation mode doesn't correctly declare required_input_distribution

3 participants