Skip to content

perf: evaluate complete partitions of aggregate windows directly in BoundedWindowAggExec - #27462

Closed
comphead wants to merge 2 commits into
apache:mainfrom
comphead:fix-26541-bwag-small-partitions
Closed

comphead wants to merge 2 commits into
apache:mainfrom
comphead:fix-26541-bwag-small-partitions

Conversation

@comphead

@comphead comphead commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

In Sorted mode, BoundedWindowAggExec builds a key, a buffered batch and a state per window expression for every partition, and prunes them again. When partitions hold one or a few rows, this bookkeeping dominates. The issue has a reproducer.

What changes are included in this PR?

When every window expression is an aggregate, partitions that start and end inside an input batch are evaluated with WindowExpr::evaluate on their slice of the batch. For an aggregate this is the calculation evaluate_stateful does for a whole partition, without the key, buffer or state. Only the first and last partition of a batch are buffered, and the results of the others are spliced in between them. A stream with a WindowStateObserver keeps the current path, since it needs the state of every partition. The per-partition evaluation moves from WindowAggExec into a helper in windows/utils.rs that both operators use.

1M rows, target_partitions = 1, datafusion-cli (release-nonlto) main PR
max(v) OVER (PARTITION BY k ORDER BY i), one row per partition 0.61 s 0.33 s
the same with 10 carried string columns 0.93 s 0.56 s
sum(v) over ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING, one row per partition 0.56 s 0.28 s

In the window_query_sql benchmark, window partition and order by, u64_wide, aggregate functions goes from 443 ms to 318 ms, and the other cases stay within noise.

What is the testing strategy for this PR?

  • A window.slt case at batch_size = 4, where a batch starts inside a buffered partition, holds a complete one and ends inside another.
  • The observer exactly-once test gets a partition inside a batch, which must still be observed.
  • A UDWF test pins the evaluator calls for a partition inside a batch.

Are there any user-facing changes?

No. Known limitation: non-aggregate window functions keep the current path. Their evaluate calls other PartitionEvaluator methods than the stream does, and evaluating each partition with evaluate_stateful made row_number and lag slower.

…BoundedWindowAggExec`

In `Sorted` mode, when every window expression is an aggregate, partitions
that start and end inside an input batch are evaluated with
`WindowExpr::evaluate` on their slice of the batch, without a key, a
buffered batch or a state per partition.
@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Nov 15, 2026
Move `compute_window_aggregates` to `windows/utils.rs` and use it in both
window operators. Splice the results of complete partitions in one pass,
drain the middle ranges in place, and share the aggregate check with
`limit_effect`.
@comphead

Copy link
Copy Markdown
Contributor Author

Superseded by #27466, which contains these two commits and extends the same path to first_value, last_value and nth_value.

@comphead comphead closed this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate performance Make DataFusion faster physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant