You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Median aggregation using DataFrame panics: "AggregateState is not a scalar aggregate" #3105
Describe the bug
I am trying to use the new exact median aggregation function introduced by @andygrove in #3009, but when I try it using the DataFrame API the operation panics. Apologies that I didn't get around to testing this while the PR was open!
To Reproduce
Here is a test case that can be run inside the src/core/tests/dataframe.rs file:
thread 'exact_median' panicked at 'unexpected accumulator state in hash aggregate: Internal("AggregateState is not a scalar aggregate")', datafusion/core/src/physical_plan/aggregates/hash.rs:434:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'exact_median' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ExternalError(Execution("Join Error: task 29 panicked")))', datafusion/core/tests/dataframe.rs:459:38
I can also reproduce this with datafusion-cli, which I would guess is the same issue?
❯ create table cpu (host string, usage float) asselect*from (values ('host0', 90.1), ('host1', 90.2));
0 rows inset. Query took 0.011 seconds.
❯ select host, median(usage) from cpu group by host;
thread 'tokio-runtime-worker' panicked at 'unexpected accumulator state in hash aggregate: Internal("AggregateState is not a scalar aggregate")', /Users/cwolff/workspace/github.com/apache/arrow-datafusion/datafusion/core/src/physical_plan/aggregates/hash.rs:517:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
ArrowError(ExternalError(Execution("Join Error: task 17 panicked")))
changed the title [-]Median aggregation using DataFrame panics[/-][+]Median aggregation using DataFrame panics: "AggregateState is not a scalar aggregate"[/+]on Dec 2, 2022
Strangely, if I run the function directly on values without first making a table it works fine.
DataFusion CLI v14.0.0
❯ select median(column2) from (values ('host0', 90.1), ('host1', 90.2));
+-----------------+
| MEDIAN(column2) |
+-----------------+
| 90.15 |
+-----------------+
1 row in set. Query took 0.053 seconds.
❯ create table cpu (host string, usage float) as select * from (values ('host0', 90.1), ('host1', 90.2));
0 rows in set. Query took 0.005 seconds.
❯ select host, median(usage) from cpu group by host;
thread 'tokio-runtime-worker' panicked at 'unexpected accumulator state in hash aggregate: Internal("AggregateState is not a scalar aggregate")', /Users/alamb/Software/arrow-datafusion2/datafusion/core/src/physical_plan/aggregates/hash.rs:517:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
ArrowError(ExternalError(Context("Join Error", External(JoinError::Panic(Id(51), ...)))))
Activity
andygrove commentedon Aug 12, 2022
Thanks @jonmmease I will take a look
wolffcm commentedon Nov 29, 2022
I can also reproduce this with
datafusion-cli, which I would guess is the same issue?alamb commentedon Dec 1, 2022
We are hitting this with IOx so I plan to fix it
[-]Median aggregation using DataFrame panics[/-][+]Median aggregation using DataFrame panics: "AggregateState is not a scalar aggregate"[/+]alamb commentedon Dec 2, 2022
Strangely, if I run the function directly on
valueswithout first making a table it works fine.alamb commentedon Dec 2, 2022
I think this code will basically happen any time median is used in a query that has more than one partition
alamb commentedon Dec 2, 2022
Proposed fix in #4488