Skip to content

Follow-on work for median aggregate function #3040

Closed

Description

@andygrove

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Follow-on issue to address feedback in #3009

In particular:

  • Run some performance test and see if it would be better to use builders in the aggregate state rather than arrays. Should we just use ScalarValue::List instead?
  • Should we have a configurable limit on the number of values that can be collected to prevent OOM?
  • Can we leverage the concat and take kernels?
  • Should we add an average kernel to arrow-rs?
  • Can we reduce memory overhead with some form of compression (such as run-length encoding or maintaining a map of unique values with counts)

Describe the solution you'd like
See above

Describe alternatives you've considered
None

Additional context
None

Activity

  1. jhorstmann commented on Aug 9, 2022

    @jhorstmann
    Contributor

    Another optimization is that you do not need to to sort all values, only the value at the middle index needs to be at the right place. Vec::select_nth_unstable does exactly that in O(n) using a quickselect algorithm.

  2. Dandandan commented on Aug 23, 2023

    @Dandandan
    Contributor

    #7376 solves most of these

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions