Skip to content

[R] dplyr::compute should convert from grouped arrow_dplyr_query to arrow Table #32973

Closed

Description

@asfimport

It is expected that dplyr::compute() will perform the calculation on the arrow dplyr query and convert it to a Table, but it does not seem to work correctly for grouped arrow dplyr queries and does not result in a Table.

mtcars |> arrow::arrow_table() |> dplyr::group_by(cyl) |> dplyr::compute() |> class()
#> [1] "arrow_dplyr_query"
mtcars |> arrow::arrow_table() |> dplyr::group_by(cyl) |> dplyr::ungroup() |> dplyr::compute() |> class()
#> [1] "Table"        "ArrowTabular" "ArrowObject"  "R6"

as_arrow_table() works fine.

mtcars |> arrow::arrow_table() |> dplyr::group_by(cyl) |> class()
#> [1] "arrow_dplyr_query"
mtcars |> arrow::arrow_table() |> dplyr::group_by(cyl) |> dplyr::compute() |> class()
#> [1] "arrow_dplyr_query"
mtcars |> arrow::arrow_table() |> dplyr::group_by(cyl) |> dplyr::collect(FALSE) |> class()
#> [1] "arrow_dplyr_query"
mtcars |> arrow::arrow_table() |> dplyr::group_by(cyl) |> arrow::as_arrow_table() |> class()
#> [1] "Table"        "ArrowTabular" "ArrowObject"  "R6"

It seems to revert to arrow dplyr query in the following line.

df <- as_adq(df)
df$group_by_vars <- query$group_by_vars
df$drop_empty_groups <- query$drop_empty_groups

 

Reporter: SHIMA Tatsuya / @eitsupi
Assignee: SHIMA Tatsuya / @eitsupi

Related issues:

PRs and other links:

Note: This issue was originally created as ARROW-17738. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Sep 16, 2022

    @asfimport
    Author

    Neal Richardson / @nealrichardson:
    Could you clarify what is not working?

  2. asfimport commented on Sep 17, 2022

    @asfimport
    Author

    SHIMA Tatsuya / @eitsupi:
    I have updated the description.
    Grouped arrow dplyr queries are not converted to tables by dplyr::compute.

  3. asfimport commented on Sep 17, 2022

    @asfimport
    Author

    SHIMA Tatsuya / @eitsupi:
    Ah, is this the intended behavior?
    I didn't understand why this behavior was intended, I think compute should return a Table here, just as dbplyr and dtplyr do.

  4. asfimport commented on Sep 17, 2022

    @asfimport
    Author

    Neal Richardson / @nealrichardson:
    They are evaluated and converted to Tables, but then if there are groups, group_by is called on the Table, which results in an arrow_dplyr_query object containing the Table. So, yes, this was intentional. Do you have a use case where this is a problem?

  5. asfimport commented on Sep 17, 2022

    @asfimport
    Author

    SHIMA Tatsuya / @eitsupi:
    I think it is confusing to users when compute does not result in a Table as intended when the group is left after summarise, etc. is executed.

    mtcars |> arrow::arrow_table() |> dplyr::group_by(vs, am) |> dplyr::summarise(wt = mean(wt)) |> dplyr::compute()
    #> Table (query)
    #> vs: double
    #> am: double
    #> wt: double
    #>
    #> * Grouped by vs
    #> See $.data for the source Arrow object
  6. asfimport commented on Oct 7, 2022

    @asfimport
    Author

    Dewey Dunnington / @paleolimbot:
    Issue resolved by pull request 14160
    #14160

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions