Repository navigation
Support window functions with empty OVER clause #298
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Window functions are a very valuable feature to have for DataFusion, as it allows to do analytical queries and things like deduplication to happen.
Describe the solution you'd like
Initial support for window functions with OVER () clause. This allows us to gradually add more features, like support for PARTITION BY and ORDER BY.
Describe alternatives you've considered
A more complete implementation in one go. Window functions however are a big feature and we probably need quite some iterations to get it in shape.
Additional context
Some material:
http://www.vldb.org/pvldb/vol8/p1058-leis.pdf&ved=2ahUKEwj80-3OjrfwAhUJPOwKHfdRAssQFjAMegQIEhAC&usg=AOvVaw2KKUPeYhyc-pEFTmlqyboj
Activity
jimexist commentedon May 13, 2021
@Dandandan i wonder how window function shall fit into the logical planner? it shall be folded into aggregate functions? but the actual output schema might be different.
Dandandan commentedon May 13, 2021
@jimexist I think a new operator might be the best to do here. They have very different semantics and needs.
Between the operators there probably is still a lot of opportunity for reuse and in the planner we can use different operators (repartition, merge, sort, etc.)
What do you think?
@jorgecarleitao @alamb
alamb commentedon May 13, 2021
I agree that a new operator (along with a new kind of aggregate) is probably best here -- the way window aggregate functions are applied is different than "normal" aggregates -- among other things for a window function there is typically one row of output for each row of input whereas normal aggregates produce one row of output for each distinct value of grouping keys.
jimexist commentedon May 13, 2021
See #334 for a general idea of how much code change is needed if we'd add window function side by side with aggregates
jimexist commentedon May 19, 2021
We'd like to support window function in three or more steps:
OVERclause #298 empty over clause (this one)jimexist commentedon May 26, 2021
row_number#375 to add window function support, streaming, androw_numberfirst_value,last_value, andnth_valuebuilt-in window functions #403first_value,last_value, andnth_valueleadandlagjimexist commentedon May 26, 2021
@alamb and @Dandandan feel free to review in order