Repository navigation
Conversation
first_value, last_value and nth_value in one callBoundedWindowAggExec
kumarUjjawal
left a comment
There was a problem hiding this comment.
Thank you @comphead
Left few comments please take a look.
| let n_partitions = partition_points.len(); | ||
| if self.can_evaluate_complete && n_partitions > 2 { | ||
| let complete = 1..n_partitions - 1; | ||
| self.complete_out = compute_window_aggregates( |
There was a problem hiding this comment.
Complete-partition results pass from evaluate_partition_batches to calculate_out_columns through the mutable SortedSearch::complete_out field. Correct output relies on unstated rules: exactly one calculate_out_columns call per batch, and the last entry of every expression's state map being that batch's last partition.
There was a problem hiding this comment.
Both rules hold by construction, and they are now stated and checked. The stream calls update_partition_batch and compute_aggregates back to back for every batch in poll_next_inner, so calculate_out_columns takes the results of the batch that set them. Every state map ends with the batch's last partition, because evaluate_stateful inserts a state for each entry of partition_buffers in order (aggregate_evaluate_stateful and StandardWindowExpr::evaluate_stateful both insert before any skip), and that partition is a new key at the end of partition_buffers. The doc of complete_out now says this, and two debug_assert!s check it on every batch that takes this path.
| let mut out_cols = partition_window_agg_states | ||
| .values() | ||
| .map(|WindowState { state, .. }| &state.out_col); | ||
| let last = complete.and_then(|_| out_cols.next_back()); |
There was a problem hiding this comment.
get_aggregate_result_out_column places complete before the last partition by taking next_back() from each expression's IndexMap. This assumes every expression's map ends with the same partition, the current batch's last one, which nothing enforces.
There was a problem hiding this comment.
Same invariant as in the thread above. calculate_out_columns now asserts that every state map ends with the last entry of partition_buffers before the splice.
| } | ||
|
|
||
| /// See [`SortedSearch::can_evaluate_complete`] | ||
| fn can_evaluate_complete(expr: &Arc<dyn WindowExpr>) -> bool { |
There was a problem hiding this comment.
can_evaluate_complete decides eligibility by downcasting to concrete types and repeats the capability check that StandardWindowExpr::evaluate_partitions already does, instead of asking the WindowExpr trait.
There was a problem hiding this comment.
The downcast follows limit_effect in the same file, which asks the same question about the concrete expression types. The flag is checked in both places on purpose: the gate runs once per stream, and evaluate_partitions has to stay correct for any caller, since WindowAggExec calls it for every expression. A WindowExpr capability method would remove the second check, but it would add public API to save one line, so I kept this.
| // Only the first partition, which may continue a buffered one, and the | ||
| // last, which may continue in the next batch, need to be buffered. | ||
| let n_partitions = partition_points.len(); | ||
| if self.can_evaluate_complete && n_partitions > 2 { |
There was a problem hiding this comment.
The optimization is all-or-nothing: one unsupported expression (row_number, rank, lag, or any UDWF without the flag) in the same window node turns it off for every expression.
There was a problem hiding this comment.
Yes, by design. The complete partitions are drained from partition_buffers, which all expressions of the node share, so every expression has to be able to evaluate them directly. A mixed node keeps the current path at the same cost: row_number next to last_value(s) IGNORE NULLS on the #27463 reproducer takes 5.3 s both on main and with this PR. Supporting mixed nodes needs a per-expression buffer, which is a bigger change than this PR.
| batch: &RecordBatch, | ||
| partitions: &[Range<usize>], | ||
| ) -> Result<Option<ArrayRef>> { | ||
| let evaluator = self.expr.create_evaluator()?; |
There was a problem hiding this comment.
StandardWindowExpr::evaluate_partitions calls create_evaluator() on every call only to check two flags, then throws the evaluator away when the function does not qualify.
There was a problem hiding this comment.
In BoundedWindowAggExec this only runs for expressions that passed the gate, so the evaluator is used. In WindowAggExec, compute_window_aggregates now stops asking an expression after its first None and evaluates it one partition at a time, so a function that does not qualify creates one extra evaluator per input.
| /// the same calls as one that is buffered | ||
| #[tokio::test] | ||
| async fn test_udwf_bounded_window_complete_partition() { | ||
| let test_state = TestState::new().with_supports_bounded_execution(); |
There was a problem hiding this comment.
test_udwf_bounded_window_complete_partition never exercises the new path. odd_counter neither sets evaluates_frames_independently nor uses_window_frame, so can_evaluate_complete is false and the query is buffered exactly as before.
There was a problem hiding this comment.
Right, and that is what it pins: a function that is neither an aggregate nor sets the flag keeps the buffered path for a partition inside a batch. Admitting every window function in can_evaluate_complete makes it fail, because evaluate would then call evaluate_all for that partition. I reworded its doc to say this. The direct path is covered by the complete_partitions case in window.slt.
| statement ok | ||
| set datafusion.execution.batch_size = 8; | ||
|
|
||
| query IIIIIIIII |
There was a problem hiding this comment.
complete_partitions case does not assert its plan (EXPLAIN). Nothing checks that the query runs through BoundedWindowAggExec in Sorted mode with 8-row input batches that actually contain complete partitions.
There was a problem hiding this comment.
Added an EXPLAIN of the same query, which shows BoundedWindowAggExec in Sorted mode over the SortExec. The results check the rest: injecting faults into the complete-partition path (frames that leave their partition, a stale frame or GROUPS state, the splice order, the first partition evaluated directly) makes this case fail.
| let arrays = arrays.iter().map(|a| a.as_ref()).collect::<Vec<_>>(); | ||
| results[idx] = Some(concat(&arrays)?); | ||
| } | ||
| Ok(results.into_iter().flatten().collect()) |
There was a problem hiding this comment.
compute_window_aggregates builds a Vec<Option<ArrayRef>> and returns it through .flatten(). Any slot left None would be dropped silently instead of raising an error.
There was a problem hiding this comment.
Rewrote it so that every expression gets exactly one concatenated array, with no Option slots.
| let mut partition_order_bys = Vec::with_capacity(order_bys.len()); | ||
| let mut ranges = Vec::with_capacity(num_rows); | ||
| for partition in partitions { |
There was a problem hiding this comment.
Bound temporary frame storage in WindowAggExec. When called from WindowAggExec, this allocates frame ranges for the entire accumulated input, rather than releasing them after each partition as before.
There was a problem hiding this comment.
Agree with @kumarUjjawal, and I measured it: on 8M rows with an INT argument and 1000-row partitions, first_value(v) OVER (PARTITION BY k ORDER BY i ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) peaks at 1006 MB RSS on the PR against 845 MB on main. That is about 20 bytes per row of frame ranges and take indices, and WindowAggExec reserves none of it (#26707). Evaluating runs of partitions of at most batch_size rows per evaluate_partitions call would keep this batch-sized. Fine as a follow-up, since an input without PARTITION BY already allocates ranges for every row on main.
There was a problem hiding this comment.
Fixed here rather than in a follow-up. compute_window_aggregates now gives evaluate_partitions runs of at most batch_size rows, or one larger partition, and WindowAggExec passes its batch_size. On the query jayzhan211 measured (8M rows, 1000-row partitions, first_value to UNBOUNDED FOLLOWING), peak RSS goes from 962 MB to 803 MB, the same as main (804 MB). The extra concatenation costs at most about 3% for a struct argument and nothing measurable for an INT.
| /// Returns `true` if, until [`Self::memoize`] is called, the result of | ||
| /// [`Self::evaluate_ranges`] for a frame depends only on `values` and that | ||
| /// frame. A new evaluator can then evaluate the frames of several small | ||
| /// partitions in one call. |
There was a problem hiding this comment.
Should we document that results must remain equivalent when partition arrays are concatenated and frame indices are rebased; otherwise UDWF authors can satisfy the stated contract and still obtain incorrect results.
There was a problem hiding this comment.
Agree with @kumarUjjawal: "depends only on values and that frame" still allows reading rows of values outside the frame, which breaks once values holds several partitions. Maybe:
/// Returns `true` if, until [`Self::memoize`] is called, the result of
/// [`Self::evaluate_ranges`] for a frame depends only on the rows of
/// `values` inside that frame. A new evaluator can then evaluate the
/// frames of several partitions, concatenated in `values`, in one call.
There was a problem hiding this comment.
Adopted the wording suggested in this thread. NthValueEvaluator reads only the validity bits inside the frame and gathers one row of it, so it meets the stricter contract.
jayzhan211
left a comment
There was a problem hiding this comment.
@comphead thanks for working on this! One optional note on the description: the Rationale opens with the mechanism, so a reviewer has to reach the table to see the gain. Maybe lead with the symptom from the two issues:
BoundedWindowAggExecis slow when PARTITION BY groups hold one or a few rows. On 1M one-row partitions,max(v) OVER (PARTITION BY k ORDER BY i)takes 0.63 s, against 0.21 s for the same values inWindowAggExec(#26541). In Comet, a Sparklast(x, ignoreNulls)window with about 2 rows per partition costs 36 s per task of 5M rows (#27463).The cost is per partition: in
Sortedmode every partition gets a key, a buffered batch and a state per window expression, andfirst_value,last_valueandnth_valuegather each partition's result with a separatetake.
| partitions: &[Range<usize>], | ||
| ) -> Result<Option<ArrayRef>> { | ||
| let evaluator = self.expr.create_evaluator()?; | ||
| if !(evaluator.uses_window_frame() && evaluator.evaluates_frames_independently()) |
There was a problem hiding this comment.
@comphead thanks for this! No test fails if this check drops evaluates_frames_independently(): a user-defined function that uses the window frame without the flag would then get one evaluate_ranges call over all partitions of a WindowAggExec input, and all user_defined_window_functions tests and window*.slt still pass. Could you add a test that pins one partition per call? This one gives 7 on the PR and 10 with the flag check removed:
Test (passes on the PR, fails with the flag check removed)
@@ struct TestState {
evaluate_all_with_rank_called: AtomicUsize,
+ /// The most rows that `evaluate` saw in `values`
+ max_values_len: AtomicUsize,
@@ impl PartitionEvaluator for OddCounter {
self.test_state.inc_evaluate_called();
+ self.test_state
+ .max_values_len
+ .fetch_max(values[0].len(), Ordering::SeqCst);
@@
+/// A function that uses the window frame without
+/// `evaluates_frames_independently` sees one partition per call
+#[tokio::test]
+async fn test_udwf_window_frame_one_partition_per_call() {
+ let test_state = TestState::new().with_uses_window_frame();
+ let TestContext { ctx, test_state } = TestContext::new(test_state);
+ execute(&ctx, "SET datafusion.execution.target_partitions = 1")
+ .await
+ .unwrap();
+
+ let sql = "SELECT x, y, val, \
+ odd_counter(val) OVER (PARTITION BY x ORDER BY y \
+ ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) \
+ from t ORDER BY x, y";
+ execute(&ctx, sql).await.unwrap();
+
+ // x = 2 has 7 of the 10 rows
+ assert_eq!(test_state.max_values_len.load(Ordering::SeqCst), 7);
+}There was a problem hiding this comment.
Confirmed: with the flag check removed, every user_defined_window_functions test and every window*.slt file still passed. Instead of a new test, test_udwf_bounded_window now asserts it. It already runs odd_counter with uses_window_frame and without the flag in WindowAggExec, so it only needed target_partitions = 1 and the max_values_len counter from your snippet. It sees 7 rows on the PR and fails with 10 when the check is removed.
| .iter() | ||
| .map(|column| column.slice(start, partition.len())), | ||
| ); | ||
| // Only the state of a GROUPS frame is specific to a partition |
There was a problem hiding this comment.
Optional cleanup: this comment could say why RANGE needs no reset, since that relies on its comparator cache being keyed by the ORDER BY arrays:
| // Only the state of a GROUPS frame is specific to a partition | |
| // A GROUPS frame keeps the group boundaries of the previous partition. | |
| // A RANGE frame caches only a comparator keyed by the ORDER BY arrays, | |
| // which are new slices for each partition. |
…BoundedWindowAggExec` In `Sorted` mode, when every window expression is an aggregate, partitions that start and end inside an input batch are evaluated with `WindowExpr::evaluate` on their slice of the batch, without a key, a buffered batch or a state per partition.
Move `compute_window_aggregates` to `windows/utils.rs` and use it in both window operators. Splice the results of complete partitions in one pass, drain the middle ranges in place, and share the aggregate check with `limit_effect`.
… `nth_value` in one call `PartitionEvaluator::evaluates_frames_independently` marks evaluators whose `evaluate_ranges` result for a frame depends only on the frame, which `first_value`, `last_value` and `nth_value` set. For them, `WindowExpr::evaluate_partitions` computes the frames of every row of a run of complete partitions and evaluates them with one `evaluate_ranges` call, so one `take` per batch instead of one per partition. `BoundedWindowAggExec` now evaluates the complete partitions of these functions directly, and `WindowAggExec` uses the same call for all partitions of its input.
`evaluate_frames` slices the batch to the rows of the partitions it evaluates, so the arguments of the first and last partition of a batch, which the stream evaluates again, are no longer evaluated twice. One `WindowFrameContext` and one vector of ORDER BY slices serve all partitions, and only the state of a GROUPS frame is reset for each one. The two `window.slt` cases for complete partitions are merged, keeping the sliding aggregate of the first.
…view `compute_window_aggregates` gives `evaluate_partitions` runs of at most `batch_size` rows, so `WindowAggExec` no longer allocates the frames of its whole input at once, and every expression gets exactly one concatenated array. The rules for splicing the results of complete partitions are stated in the docs and checked with `debug_assert!`. `test_udwf_bounded_window` checks that a function without `evaluates_frames_independently` sees one partition per call, and the `window.slt` case asserts its plan.
ed7a1f7 to
6b8ef8e
Compare
|
Thanks for the reviews. Rebased on |
|
Thanks @kumarUjjawal @jayzhan211 for the review, please take another look |
| ); | ||
| let complete = 1..n_partitions - 1; | ||
| // One run, since these partitions are inside one input batch | ||
| self.complete_out = compute_window_aggregates( |
There was a problem hiding this comment.
complete_out is set in evaluate_partition_batches before anything else in the batch can fail. If the batch then errors, nothing clears it.
There was a problem hiding this comment.
True, but the stream cannot resume after an error on main either, so I kept it. update_partition_batch can fail in extend or concat_batches after earlier partitions were already appended to partition_buffers. compute_aggregates can fail on the second window expression after the first one advanced its state. Polling again after such an Err would emit misaligned rows in both cases, and consumers stop at the first error. If we want the stream to be resumable, that is a wider change than complete_out.
| assert_eq!(test_state.evaluate_all_called(), 0); | ||
| } | ||
|
|
||
| /// A window function that is neither an aggregate nor sets |
There was a problem hiding this comment.
Myabe a test for evaluates_frames_independently flag, upto you.
There was a problem hiding this comment.
Added in e0d1948 by reusing test_udwf_bounded_window, which now runs with the flag off and on. With it on, WindowAggExec evaluates both partitions in one call, so evaluate sees all 10 rows instead of 7, with the same results.
| results[idx].push(result); | ||
| } | ||
| } | ||
| if !per_partition.is_empty() { |
There was a problem hiding this comment.
For aggregates, the new per-partition loop slices a RecordBatch and calls evaluate once per partition. Each call re-evaluates the argument, FILTER and ORDER BY expressions and builds a new WindowFrameContext and comparator for a tiny slice. evaluate_frames instead evaluates arguments once per run.
There was a problem hiding this comment.
Confirmed. In a profile of window partition and order by, u64_wide, aggregate functions, the per-partition evaluate_args was 29% of compute_window_aggregates, order_by_columns 12%, the batch slice and its drop 9%, and the per-partition iter_to_array 7%.
Plain and sliding aggregates now implement evaluate_partitions (AggregateWindowExpr::aggregate_evaluate_partitions) in e0d1948. The arguments, FILTER and ORDER BY are evaluated once per run. Each partition gets a new accumulator, and its frames are offsets into the run's arrays, so the values are not sliced per partition. A constant result is computed once per partition and repeated with one take. A run with a single partition, which can be larger than a batch, keeps aggregate_evaluate, so its frames are not collected.
window_query_sql against main: window partition and order by, u64_wide, aggregate functions 451 ms to 96 ms (333 ms before this change), and window partition by, u64_wide, aggregate functions 223 ms to 124 ms. The other cases are within noise. The complete_partitions case in window.slt gained a FILTER column, since without it no test caught a dropped FILTER on this path.
There was a problem hiding this comment.
evaluate_frames reuses one WindowFrameContext across partitions. It resets the GROUPS state by writing its public fields by hand, and RANGE is only safe because its comparator cache is keyed by Arc::ptr_eq on fresh slices. The reset belongs in window_state.rs, as a WindowFrameContext::reset() or a new context per partition.
There was a problem hiding this comment.
Agreed, done in e0d1948. WindowFrameContext::reset() in window_state.rs clears the GROUPS state and drops the cached RANGE comparator, so RANGE no longer depends on the Arc::ptr_eq key. The frame loop moved to a shared calculate_frames helper, which the aggregate path from your other comment uses too.
I kept one context with a reset rather than a new one per partition. A new context clones the shared Arc<WindowFrame>, and that refcount contends across threads: with 1M one-row partitions it took 90 ms at target_partitions = 18 against 7 ms with the reused context.
There was a problem hiding this comment.
Should we update The PartitionEvaluator docs for the new flag.
There was a problem hiding this comment.
Done in e0d1948. The implementation table now points to evaluate_ranges and evaluates_frames_independently.
…ress review - Plain and sliding aggregates implement `evaluate_partitions`: arguments, FILTER and ORDER BY are evaluated once per run, each partition gets a new accumulator, and its frames are offsets into the run's arrays. - `WindowFrameContext::reset` replaces the hand-written GROUPS reset, and a shared `calculate_frames` helper serves the standard and aggregate paths. - Document `evaluates_frames_independently` next to the implementation table, test the flag in `test_udwf_bounded_window`, and add a FILTER column to the `complete_partitions` SLT case.
|
Thanks @kumarUjjawal. Addressed in e0d1948. Aggregates now evaluate complete partitions in one call as well, which takes |
jayzhan211
left a comment
There was a problem hiding this comment.
@comphead thanks for the update! Results match main on every shape I tried (266 runs across frame kinds, FILTER, DISTINCT and batch sizes 1 to 8192), and small partitions are 2x to 8x faster. Two comments inline: a slowdown on constant aggregates with mid-size partitions, and a test for the run limit.
| // Repeat the result of each partition for its rows | ||
| let indices = UInt32Array::from_iter_values( | ||
| partitions | ||
| .iter() | ||
| .enumerate() | ||
| .flat_map(|(idx, p)| std::iter::repeat_n(idx as u32, p.len())), | ||
| ); | ||
| Ok(take(&ScalarValue::iter_to_array(results)?, &indices, None)?) |
There was a problem hiding this comment.
The take here makes constant aggregates in WindowAggExec slower than main once partitions hold a few hundred rows or more. On 4M rows, sum(v) OVER (PARTITION BY k) with 2000-row partitions goes from 11.0 ms to 12.4 ms, and max(s) on strings from 26.6 ms to 37.4 ms (window elapsed_compute, median of 9 to 15 runs). window_query_sql misses it because its partitions are either one row or about 100k rows. Filling each partition, as main does, brings these back to 11.5 ms and 27.6 ms, and one-row partitions stay 2x faster than main (461 ms against 913 ms):
| // Repeat the result of each partition for its rows | |
| let indices = UInt32Array::from_iter_values( | |
| partitions | |
| .iter() | |
| .enumerate() | |
| .flat_map(|(idx, p)| std::iter::repeat_n(idx as u32, p.len())), | |
| ); | |
| Ok(take(&ScalarValue::iter_to_array(results)?, &indices, None)?) | |
| // Filling each partition is cheaper than a gather with one index per | |
| // row once partitions hold more than a few rows | |
| let arrays = results | |
| .iter() | |
| .zip(partitions) | |
| .map(|(value, p)| value.to_array_of_size(p.len())) | |
| .collect::<Result<Vec<_>>>()?; | |
| let arrays = arrays.iter().map(|a| a.as_ref()).collect::<Vec<_>>(); | |
| Ok(arrow::compute::concat(&arrays)?) |
The imports then need:
-use arrow::array::{Array, ArrayRef, UInt32Array, make_comparator, new_empty_array};
+use arrow::array::{Array, ArrayRef, make_comparator, new_empty_array};
use arrow::compute::SortOptions;
use arrow::compute::filter as arrow_filter;
use arrow::compute::kernels::sort::SortColumn;
-use arrow::compute::take;Measurements: main / PR / with the suggestion (4M rows, target_partitions = 1, datafusion-cli release-nonlto)
OVER (PARTITION BY k), rows per partition |
main | PR | suggestion |
|---|---|---|---|
sum(v), 1 |
913 ms | 351 ms | 461 ms |
sum(v), 4 |
235 ms | 95 ms | 124 ms |
sum(v), 100 |
20.8 ms | 16.4 ms | 16.7 ms |
sum(v), 2000 |
11.0 ms | 12.4 ms | 11.5 ms |
sum(v), 4000 |
10.9 ms | 12.3 ms | 10.9 ms |
max(s), 4 |
321 ms | 138 ms | 173 ms |
max(s), 2000 |
26.6 ms | 37.4 ms | 27.6 ms |
The suggestion gives the same results as main on 70 runs of constant aggregates (int, string, dictionary, decimal, FILTER, DISTINCT) at batch sizes 1 to 8192.
| .unwrap(); | ||
| let independent = execute(&ctx, BOUNDED_WINDOW_QUERY).await.unwrap(); | ||
| assert_eq!(batches_to_string(&independent), batches_to_string(&actual)); | ||
| assert_eq!(test_state.max_values_len(), 10); |
There was a problem hiding this comment.
No test fails if compute_window_aggregates drops the batch_size limit on its runs (partition_runs(partitions, usize::MAX)), so the memory fix from the last round could come undone without notice. This passes on the PR and fails with 10 instead of 7 when the limit is removed:
| assert_eq!(test_state.max_values_len(), 10); | |
| assert_eq!(test_state.max_values_len(), 10); | |
| // A call gets at most `batch_size` rows, or one larger partition, so that | |
| // the frames of `WindowAggExec` stay batch-sized | |
| let test_state = TestState { | |
| evaluates_frames_independently: true, | |
| ..TestState::new().with_uses_window_frame() | |
| }; | |
| let TestContext { ctx, test_state } = TestContext::new(test_state); | |
| execute(&ctx, "SET datafusion.execution.target_partitions = 1") | |
| .await | |
| .unwrap(); | |
| execute(&ctx, "SET datafusion.execution.batch_size = 4") | |
| .await | |
| .unwrap(); | |
| let bounded = execute(&ctx, BOUNDED_WINDOW_QUERY).await.unwrap(); | |
| assert_eq!(batches_to_string(&bounded), batches_to_string(&actual)); | |
| assert_eq!(test_state.max_values_len(), 7); |
Which issue does this PR close?
BoundedWindowAggExec: evaluate small partitions of window functions such aslast_valuewithout per-partition state #27463.BoundedWindowAggExechas a high fixed cost per partition when partitions are small #26541.Rationale for this change
BoundedWindowAggExecis slow when PARTITION BY groups hold one or a few rows. On 1M one-row partitions,max(v) OVER (PARTITION BY k ORDER BY i)takes 0.63 s, against 0.21 s for the same values inWindowAggExec(#26541). In Comet, a Sparklast(x, ignoreNulls)window with about 2 rows per partition costs 36 s per task of 5M rows (#27463).The cost is per partition: in
Sortedmode every partition gets a key, a buffered batch and a state per window expression, andfirst_value,last_valueandnth_valuegather each partition's result with a separatetake.What changes are included in this PR?
Partitions that start and end inside an input batch are evaluated directly, with one new
WindowExpr::evaluate_partitionscall, when every window expression is an aggregate or one of these three functions. Only the first and last partition of a batch are buffered, and the results of the others are spliced in between them. A stream with aWindowStateObserverkeeps the current path, since it needs the state of every partition.first_value,last_valueandnth_valueset a newPartitionEvaluator::evaluates_frames_independentlyflag. The frames of every row of these partitions are evaluated with oneevaluate_rangescall, so there is onetakeper batch.WindowAggExecshares this per-partition evaluation, so it gets the single call too, on runs of at mostbatch_sizerows so that the frames stay batch-sized.datafusion-cli(release-nonlto),target_partitions = 1, windowelapsed_computemainmax(v) OVER (PARTITION BY k ORDER BY i), 1M rows, one row per partitionlast_value(s) IGNORE NULLSon a struct, 10 carried columnslast_value(s) IGNORE NULLStoUNBOUNDED FOLLOWING, inWindowAggExecIn the
window_query_sqlbenchmark,window partition and order by, u64_widegoes from 451 ms to 96 ms with aggregate functions and from 427 ms to 33 ms with built-in functions.window partition by, u64_widegoes from 223 ms to 124 ms with aggregate functions and from 238 ms to 8 ms with built-in functions. The other cases stay within noise. Peak memory ofWindowAggExecstays as onmain.What is the testing strategy for this PR?
window.sltcase atbatch_size = 8with complete partitions in both batches. The second batch starts inside a buffered partition and ends inside another, ORDER BY values repeat across partitions, and NULLs meet ROWS, RANGE and GROUPS frames next to plain and sliding aggregates and a FILTER. AnEXPLAINshows that it runs inBoundedWindowAggExecinSortedmode.Are there any user-facing changes?
No.
PartitionEvaluator::evaluates_frames_independently,WindowExpr::evaluate_partitionsandAggregateWindowExpr::aggregate_evaluate_partitionsare new provided methods, andWindowFrameContext::resetis new. Other window functions, such asrow_number,rankandlag, and user-defined functions without the flag keep the current path.