Repository navigation
Description
{}Proposed approach{}:
- add functionality to allow binding registration with the
pkg::fun()name;- Modify register_binding() to register 2 identical copies for each pkg::fun binding, fun and pkg::fun.
- Add a binding for the :: operator, which helps with retrieving bindings from the function registry.
- Add generic unit tests for the pkg::fun functionality.
- register
nse_funcsrequiring indirect mapping- register each binding with and without the pkg:: prefix
- add / update unit tests for the nse_funcs bindings to include at least one pkg::fun() call for each binding
- register
nse_funcsrequiring direct mapping (unary and binary bindings)- register each binding with and without the pkg:: prefix
- add / update unit tests for the nse_funcs bindings to include at least one pkg::fun() call for each binding
- register
agg_funcsfor use withsummarise() - document changes in the Writing bindings documentation
-
going forward we should be using pkg::fun when defining a binding, which will register 2 copies of the same binding.
Different implementation options are outlined and discussed in the design document.
{}Description{}:
Currently we implement a number of functions from packages likelubridatewhich work well when called without namespacing (e.g.{}year(){}), however if someone callslubridate::year()we get a not-implemented method (e.g.{}Warning: Expression lubridate::year(time_hour) not supported in Arrow{}). Is it possible for us to look and see if we have an arrow function that matches the function itself.
{code:r}
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)ds <- InMemoryDataset$create(nycflights13::flights)
ds %>%
mutate(year = lubridate::year(time_hour)) %>%
collect()
#> Warning: Expression lubridate::year(time_hour) not supported in Arrow; pulling
#> data into R
#> # A tibble: 336,776 × 19
#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
#>
#> 1 2013 1 1 517 515 2 830 819
#> 2 2013 1 1 533 529 4 850 830
#> 3 2013 1 1 542 540 2 923 850
#> 4 2013 1 1 544 545 -1 1004 1022
#> 5 2013 1 1 554 600 -6 812 837
#> 6 2013 1 1 554 558 -4 740 728
#> 7 2013 1 1 555 600 -5 913 854
#> 8 2013 1 1 557 600 -3 709 723
#> 9 2013 1 1 557 600 -3 838 846
#> 10 2013 1 1 558 600 -2 753 745
#> # … with 336,766 more rows, and 11 more variables: arr_delay ,
#> # carrier , flight , tailnum , origin , dest ,
#> # air_time , distance , hour , minute , time_hour
-
ds %>%
mutate(year = year(time_hour)) %>%
collect()
#> # A tibble: 336,776 × 19
#> year month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
#>
#> 1 2013 1 1 517 515 2 830 819
#> 2 2013 1 1 533 529 4 850 830
#> 3 2013 1 1 542 540 2 923 850
#> 4 2013 1 1 544 545 -1 1004 1022
#> 5 2013 1 1 554 600 -6 812 837
#> 6 2013 1 1 554 558 -4 740 728
#> 7 2013 1 1 555 600 -5 913 854
#> 8 2013 1 1 557 600 -3 709 723
#> 9 2013 1 1 557 600 -3 838 846
#> 10 2013 1 1 558 600 -2 753 745
#> # … with 336,766 more rows, and 11 more variables: arr_delay ,
#> # carrier , flight , tailnum , origin , dest ,
#> # air_time , distance , hour , minute , time_hour
{code}
Reporter: Jonathan Keane / @jonkeane
Assignee: Dragoș Moldovan-Grünfeld / @dragosmg
Subtasks:
- [R] Add functionality to allow binding registration with the
pkg::funname - [R] Register
nse_funcsrequiring indirect mapping - [R] Register
nse_funcsrequiring direct mapping (unary & binary) - [R] [Doc] Document changes in the Writing bindings documentation
Related issues:
- [R] dplyr
nfunction cannot be called withdplyr::n()(is duplicated by) - [R] Create a function registry for our NSE funcs (relates to)
PRs and other links:
Note: This issue was originally created as ARROW-14575. Please see the migration documentation for further details.
Activity
asfimport commentedon Nov 3, 2021
Neal Richardson / @nealrichardson:
We could pre-process the expression/quosure and remove any
::, though that might not be safe? We could also include an allowlist/mapping of functions we would do this for (e.g.list(lubridate = c("year", "month", ...), dplyr = c("case_when", "n", ...), stringr = c("str_stuff", ...)).asfimport commentedon Nov 3, 2021
Jonathan Keane / @jonkeane:
Yeah, there is certainly some edgecases/unsafeness that we need to consider. Though hard-coding is frustrating, the allowlist would make it explicit what our intention is with the binding. I wonder if it is possible to mask with the {pkg::} included so that we would get the right behavior if someone had a custom-defined function that collides with a function we have made a binding for (e.g. I've defined a custom
year()in my code, I know that that will mask lubridate'syear(), so I putlubridate::year()in all my pipelines, but sometimes need to / want to call my custom year — this wouldn't (possibly) work until ARROW-14071 anyway).Getting the execution exactly right and being able to select the right
year()here is probably less important than erroring loudly and not using the bindings version when that's not what dplyr would do in that case. Though I will say that I have definitely seen things like this and worse in actuality!asfimport commentedon Nov 3, 2021
Dewey Dunnington / @paleolimbot:
It might be nice for clarity to register the package name with the function (i.e.,
arrow_register_translation("year", function(...) {}, package = "lubridate")instead ofnse_funcs$year <- function(...) {}.Erroring out could look like this (only slightly better than the current error, though).
A slightly different definition of
::could ignore it with a warning, although this is possibly a bit too magical.asfimport commentedon Nov 8, 2021
Dewey Dunnington / @paleolimbot:
A few ideas related to this that may or may not help organize the increasing set of functions from different packages.
Translations for specific packages as R6 classes? Advantage here just that we get documentation that isn’t checked by CRAN for free.
...and a totally untested interface for keeping track of function translations from multiple packages.
Helper to detect the package that a user intended based on their current search path
Using R6 for my basic implementation of a function registry that includes the intended package for a translation. These can be stacked and recombined similar to how environments can be stacked and combined so that each set of translations can have a limited scope.
In practice this would be an under-the-hood thing accessed by regular functions
The basic idea is that you can register some functions and get a decent error when requesting one that doesn’t exist.
Need to override
::for the nice errorsUse
translations_with_search()to resolve the list of translations based on the current search path (so that we can use it for a data mask).asfimport commentedon May 16, 2022
Dragoș Moldovan-Grünfeld / @dragosmg:
I have an draft PR for allowing
pkg::prefixes. What do you think of this approach?"package_"before the function name. For example"as_datetime"becomes{}"lubridate_as_datetime"{}. Doing this mainly for 2 reasons:{}::{}) operator (easier dealing with strings only){}"as_datetime"{}) and its full name ({}"lubridate_as_datetime"{}){}::{}) with an underscore ({}_{})::in the error message)pkg::fun() not supported in Arrow".asfimport commentedon Jul 8, 2022
Dragoș Moldovan-Grünfeld / @dragosmg:
I went in a slightly different direction from the one sketched-out by @paleolimbot for several reasons:
asfimport commentedon Jul 15, 2022
Neal Richardson / @nealrichardson:
Issue resolved by pull request 13160
#13160