Skip to content

fix: TopK aggregation now uses indices instead of column names for computed projections - #27430

Open
vikrantmehta123 wants to merge 2 commits into
apache:mainfrom
vikrantmehta123:make-topk-aggregation-use-sort-col-index
Open

vikrantmehta123 wants to merge 2 commits into
apache:mainfrom
vikrantmehta123:make-topk-aggregation-use-sort-col-index

Conversation

@vikrantmehta123

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

This PR fixes a query correctness bug. The issue details the bug.

What changes are included in this PR?

Previously, TopKAggregation followed the sort key through projections by name and skipped computed expressions. This caused the issue that if a computed expression was aliased to the aggregate's name, the rule could not distinguish the two.

  • datafusion/physical-optimizer/src/topk_aggregation.rs: the projection arm no longer loops over every expression to match on the name. It looks up the expression at the sort column's index. If that expression is not a plain Column then the optimization won't be applied.

What is the testing strategy for this PR?

Two SLTs are added in datafusion/sqllogictest/test_files/aggregates_topk.slt. One covers the query result and the other covers the query plans. The query used in the SLTs is the one from the issue."

Tests were run before applying the fix to ensure that the bug exists and the tests catch the bug.

Are there any user-facing changes?

Yes, the query now returns correct output. No change in the public APIs.

@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Nov 14, 2026
@vikrantmehta123

Copy link
Copy Markdown
Contributor Author

Hi @jayzhan211, a question on the scope. In this PR, I changed only the ProjectionExec arm. The AggregateExec arm still passes the column by name to try_optimize_topk(). I'm not sure whether the expectation is to change both for consistency. Could you suggest whether I need to look into that as well?

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @vikrantmehta123! Optional: the Rationale could state the symptom, so a reader can see the need without opening the issue. For example:

ORDER BY <name> LIMIT n above a GROUP BY returns wrong rows, with no error, when a computed column between them reuses the aggregate's or group key's name. For example, SELECT -k AS k FROM (SELECT DISTINCT k FROM t) ORDER BY k DESC LIMIT 2 over k = 1..4 returns -3, -4 instead of -1, -2. This PR makes TopKAggregation follow the sort column through each projection by position, and stop at a computed expression.

Comment on lines +924 to +925
statement ok
DROP TABLE topk_computed_alias;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: this fix also covers SELECT DISTINCT, where the reused name is a plain column name, so it's the shape users are most likely to hit. On main this query returns -3, -4. Could you add it?

Suggested change
statement ok
DROP TABLE topk_computed_alias;
# The same bug for SELECT DISTINCT, where the reused name is a plain column name
query I
SELECT -g AS g FROM (SELECT DISTINCT g FROM topk_computed_alias)
ORDER BY g DESC
LIMIT 2;
----
-1
-2
statement ok
DROP TABLE topk_computed_alias;

Comment on lines +145 to +146
// A computed expression can produce values that aggregation did not see.
// We cannot apply the topk optimization then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional cleanup: the values aren't the problem here. The sort orders by the computed value, and that order can differ from the aggregate's, so the comment could say that:

Suggested change
// A computed expression can produce values that aggregation did not see.
// We cannot apply the topk optimization then
// The sort orders by the computed value, whose order can differ
// from the aggregate's (`-min(x)` reverses it).

@jayzhan211

Copy link
Copy Markdown
Contributor

@vikrantmehta123 No need to change the AggregateExec arm. An aggregate's output names are unique: grouping by a column named "max(t.x)" next to max(x) is rejected at planning with "Schema contains qualified field name … ambiguous". So matching by name there finds the same field that matching by index would, and keeping it avoids changing the signature of try_optimize_topk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TopK aggregation keeps the wrong groups when a computed column reuses the aggregate's name

2 participants