Repository navigation
fix: TopK aggregation now uses indices instead of column names for computed projections - #27430
vikrantmehta123 wants to merge 2 commits into
Conversation
… reusing the aggregate's name
…euses the aggregate's name
|
Hi @jayzhan211, a question on the scope. In this PR, I changed only the |
jayzhan211
left a comment
There was a problem hiding this comment.
Thanks @vikrantmehta123! Optional: the Rationale could state the symptom, so a reader can see the need without opening the issue. For example:
ORDER BY <name> LIMIT nabove aGROUP BYreturns wrong rows, with no error, when a computed column between them reuses the aggregate's or group key's name. For example,SELECT -k AS k FROM (SELECT DISTINCT k FROM t) ORDER BY k DESC LIMIT 2over k = 1..4 returns-3, -4instead of-1, -2. This PR makesTopKAggregationfollow the sort column through each projection by position, and stop at a computed expression.
| statement ok | ||
| DROP TABLE topk_computed_alias; |
There was a problem hiding this comment.
Optional: this fix also covers SELECT DISTINCT, where the reused name is a plain column name, so it's the shape users are most likely to hit. On main this query returns -3, -4. Could you add it?
| statement ok | |
| DROP TABLE topk_computed_alias; | |
| # The same bug for SELECT DISTINCT, where the reused name is a plain column name | |
| query I | |
| SELECT -g AS g FROM (SELECT DISTINCT g FROM topk_computed_alias) | |
| ORDER BY g DESC | |
| LIMIT 2; | |
| ---- | |
| -1 | |
| -2 | |
| statement ok | |
| DROP TABLE topk_computed_alias; |
| // A computed expression can produce values that aggregation did not see. | ||
| // We cannot apply the topk optimization then |
There was a problem hiding this comment.
Optional cleanup: the values aren't the problem here. The sort orders by the computed value, and that order can differ from the aggregate's, so the comment could say that:
| // A computed expression can produce values that aggregation did not see. | |
| // We cannot apply the topk optimization then | |
| // The sort orders by the computed value, whose order can differ | |
| // from the aggregate's (`-min(x)` reverses it). |
|
@vikrantmehta123 No need to change the |
Which issue does this PR close?
Rationale for this change
This PR fixes a query correctness bug. The issue details the bug.
What changes are included in this PR?
Previously,
TopKAggregationfollowed the sort key through projections by name and skipped computed expressions. This caused the issue that if a computed expression was aliased to the aggregate's name, the rule could not distinguish the two.datafusion/physical-optimizer/src/topk_aggregation.rs: the projection arm no longer loops over every expression to match on the name. It looks up the expression at the sort column's index. If that expression is not a plainColumnthen the optimization won't be applied.What is the testing strategy for this PR?
Two SLTs are added in
datafusion/sqllogictest/test_files/aggregates_topk.slt. One covers the query result and the other covers the query plans. The query used in the SLTs is the one from the issue."Tests were run before applying the fix to ensure that the bug exists and the tests catch the bug.
Are there any user-facing changes?
Yes, the query now returns correct output. No change in the public APIs.