You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently there are two ways to compute a group by. The supported way is to use an aggregate node in an exec plan. The second (internal) way is to use the internal function arrow::internal::GroupBy. This internal function simulates, but does not actually use, an aggregate node.
The internal implementation has caused issues in the past where we did not notice an error in the aggregate node's invocation of aggregate kernels since we use the internal function for testing aggregates and it behaves slightly differently. The internal implementation also requires maintenance and significantly complicated #14352 .
I would like to remove the internal implementation. Unfortunately, the internal implementation is used by tests, benchmarks, and pyarrow. However, we should be able to update those bindings to a friendly wrapper around exec plans.
I think the current _group_by interface is awkward, because it requires target "names", which are only used for function arity. But it looks like #14867 will replace that anyway.
Also, I've been using _group_by for aggregating batches, with out-of-core data. Whereas #14867 looks like it will require a table. Just something to consider, but I guess I could always wrap a single record batch into a table.
Activity
Modified arrow::internal::GroupBy to use an exec plan directly instea…
coady commentedon Feb 11, 2023
I think the current
_group_byinterface is awkward, because it requires target "names", which are only used for function arity. But it looks like #14867 will replace that anyway.Also, I've been using
_group_byfor aggregating batches, with out-of-core data. Whereas #14867 looks like it will require a table. Just something to consider, but I guess I could always wrap a single record batch into a table.Thanks.
westonpace commentedon Feb 15, 2023
Yes, wrapping a single batch in a table should be effective.
apacheGH-14866: Modified arrow::internal::GroupBy to use an exec plan…
GH-14866: [C++] Remove internal GroupBy implementation (#14867)