Repository navigation
Cherry-pick apache/datafusion#24497: stream aggregates over grouped input - #207
Merged
erik-sargent merged 2 commits intoNov 17, 2026
Conversation
Tristan1900
force-pushed
the
wenqi.mou/cherry-pick/apache-pr-24497-20261111
branch
from
November 11, 2026 23:00
cfacd91 to
2d0191d
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## wenqi.mou/cherry-pick/apache-pr-24698-20261111 #207 +/- ##
==================================================================================
- Coverage 81.21% 81.21% -0.01%
==================================================================================
Files 1112 1112
Lines 389885 390276 +391
Branches 389885 390276 +391
==================================================================================
+ Hits 316650 316956 +306
- Misses 54385 54411 +26
- Partials 18850 18909 +59 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Tristan1900
marked this pull request as ready for review
November 12, 2026 18:23
Tristan1900
force-pushed
the
wenqi.mou/cherry-pick/apache-pr-24497-20261111
branch
from
November 12, 2026 18:36
2d0191d to
5e08d7d
Compare
Tristan1900
force-pushed
the
wenqi.mou/cherry-pick/apache-pr-24497-20261111
branch
from
November 12, 2026 18:56
5e08d7d to
a541153
Compare
Tristan1900
force-pushed
the
wenqi.mou/cherry-pick/apache-pr-24497-20261111
branch
from
November 12, 2026 20:13
a541153 to
f40f3c9
Compare
This comment has been minimized.
This comment has been minimized.
geoffreyclaude
approved these changes
Nov 16, 2026
erik-sargent
force-pushed
the
wenqi.mou/cherry-pick/apache-pr-24497-20261111
branch
from
November 16, 2026 18:41
f40f3c9 to
d4fc203
Compare
Extract the upstream eligibility checks and regression tests for constant-only groups, grouping sets, and PartialReduce. Keep release-55 stream assertions and plan expectations while retaining ordering-based selection. Co-authored-by: Jay Zhan <jayzhan211@gmail.com> (partially cherry picked from commit e7c51c7)
## Which issue does this PR close? - Closes apache#24438. - Builds on apache#24737, apache#24697, and apache#24698 (all merged). ## Rationale for this change An aggregate can emit a completed group once it knows that the same complete grouping tuple cannot appear again in its input partition. `EquivalenceProperties::grouping_satisfy` now provides that guarantee for both lexicographically ordered input and explicitly grouped input. For example, these complete `(key, time_bin)` groups are contiguous even though their values are not globally sorted: ```text (A, 20), (A, 20) | (B, 20), (B, 20) | (A, 0), (A, 0) ``` The aggregate can therefore release each group as its run ends. Its clustering mode describes that guarantee, while equivalence properties describe any actual output ordering. ## What changes are included in this PR? - Query the input equivalence properties for the complete `GROUP BY` tuple during `AggregateExec` construction. - Select `GroupClusteringMode::Full` when complete grouping tuple is satisfied, at least one grouping expression is non-constant, and grouping uses one non-nulling mask. - Select clustered execution for grouped input, allowing completed groups to be emitted incrementally. - Use input group metadata for clustered execution in `Single`, `SinglePartitioned`, and `Partial` aggregations. In a two-stage plan, the partial aggregate clears explicit grouping metadata from its output, so this property alone does not enabled clustered final aggregation. The final stage can still use ordering provided by its own input. ```text input grouping: [[key, time_bin]] GROUP BY: [key, time_bin] output ordering: none grouping_satisfy(GROUP BY) -> GroupClusteringMode::Full -> EmissionType::Incremental -> ClusteredSingleAggregateStream ``` ## Stack 1. [apache#24737 — test: cover unsorted contiguous groups in one partition](apache#24737) (merged) 2. [apache#24697 — refactor: use group clustering terminology in aggregate execution](apache#24697) (merged) 3. [apache#24698 — feat: add grouping equivalence properties](apache#24698) (merged) 4. [apache#24497 — feat: stream aggregates over grouped input](apache#24497) ← **this PR** ## Are these changes tested? The characterization test from apache#24737 declares its complete grouping tuple and verifies `GroupClusteringMode::Full`, `EmissionType::Incremental`, and `ClusteredSingleAggregateStream`. It uses `BarrierExec` to hold back the second input batch until the first completed group is emitted, then checks the full result, including a group spanning both batches. Regression tests cover `PartialReduce` with ordered and explicitly grouped input, and constant-only grouping with both a literal and a column made constant by a filter. A projection test verifies that a semantically equivalent `date_bin` grouping tuple reaches `AggregateExec` and enables incremental emission. A repeated-mask regression compares grouped and unannotated inputs and verifies that both produce the same merged groups with `GroupClusteringMode::None`. ## Are there any user-facing changes? Yes. Aggregates whose complete grouping tuple is satisfied by the input equivalence properties can execute with full group clustering and incremental emission on unsorted input. ## Review this layer [View only this PR layer](xavlee/datafusion@a616831...3b42751) --------- Co-authored-by: Jay Zhan <jayzhan211@gmail.com> (cherry picked from commit e7c51c7)
erik-sargent
force-pushed
the
wenqi.mou/cherry-pick/apache-pr-24497-20261111
branch
from
November 17, 2026 14:04
d4fc203 to
a2e1e5d
Compare
erik-sargent
deleted the
wenqi.mou/cherry-pick/apache-pr-24497-20261111
branch
November 17, 2026 15:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PartialReduceon hash aggregation.