You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Dewey Dunnington / @paleolimbot:
I did a bit of looking into this...lubridate uses a custom C parser for its order-based datetime parsers. That said, its functionality can be approximated by {}coalesce(strptime(dt_string, "format1"), strptime(dt_string, "format2"), ...){}. Is it worth translating the functions with an approximation that handles most of the use cases?
Some testing that might be useful when putting together a PR:
Dewey Dunnington / @paleolimbot:
lubridate does it by ignoring all non-alpha/numeric characters so it'll happily parse something like "01:::::23::::45" just the same as "012345" or anywhere inbetween. I forget the exact incantations for strptime, but we could provide a list of formats we're willing to consider for each of them or preprocess with a regex to reduce the number of formats we need to consider (i.e., gsub("[^A-Za-z0-9.]", " ")). If that sounds close enough for you I'm happy to make a PR!
Dragoș Moldovan-Grünfeld / @dragosmg: @paleolimbot I don't think we can rely on coalesce() to iterate through the various formats supported for {}ymd(){}. It would need to rely on the assumption that the passed format matches the data or otherwise fail. Sadly, arrow works with a wrong format resulting in weird timestamps:
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(lubridate))
df<- tibble(x= c("09-01-01", "09-01-02", "09-01-03"))
df#> # A tibble: 3 × 1#> x #> <chr> #> 1 09-01-01#> 2 09-01-02#> 3 09-01-03# lubridate::ymd()df %>%
mutate(y= ymd(x))
#> # A tibble: 3 × 2#> x y #> <chr> <date> #> 1 09-01-01 2009-01-01#> 2 09-01-02 2009-01-02#> 3 09-01-03 2009-01-03# y = short year correctdf %>%
record_batch() %>%
mutate(y= strptime(x, format="%y-%m-%d", unit="us")) %>%
collect()
#> # A tibble: 3 × 2#> x y #> <chr> <dttm> #> 1 09-01-01 2009-01-01 00:00:00#> 2 09-01-02 2009-01-02 00:00:00#> 3 09-01-03 2009-01-03 00:00:00# Y = long year this should fail in order for us to rely on coalescedf %>%
record_batch() %>%
mutate(y= strptime(x, format="%Y-%m-%d", unit="us")) %>%
collect()
#> # A tibble: 3 × 2#> x y #> <chr> <dttm> #> 1 09-01-01 0008-12-31 23:58:45#> 2 09-01-02 0009-01-01 23:58:45#> 3 09-01-03 0009-01-02 23:58:45
Therefore, my early (and somewhat naive) conclusion would be that we cannot implement arrow::ymd() binding as {}coalesce(strptime(x, format1), strptime(x, format2), ...){}. What do you think?
Dragoș Moldovan-Grünfeld / @dragosmg:
We could to some processing to figure out how many characters we have (in the string to be parsed) in between the separators (or how many characters we have in total, in the cases where we have no separator) and only try with the suitable formats. i.e. in the example above not try to parse with {}%Y{}, only {}%y{}.
Dragoș Moldovan-Grünfeld / @dragosmg:
Another alternative would be for strptime to error when the selected format does not match the data (for example, attempting to parse "09-12-31" with {}"%Y-%m-%d"{}should error due to a mismatch in the length of the year). Then we could rely on this behaviour with {}coalesce{}.
Dragoș Moldovan-Grünfeld / @dragosmg: lubridate uses guess_formats() to identify the likely candidates. We could try something similar, where we have a list of supported formats (something similar to this), which we then narrow down to the most likely ones. Only then use something like {}coalesce(){}.
Activity
asfimport commentedon Nov 8, 2021
Dewey Dunnington / @paleolimbot:
I did a bit of looking into this...lubridate uses a custom C parser for its order-based datetime parsers. That said, its functionality can be approximated by
{}coalesce(strptime(dt_string, "format1"), strptime(dt_string, "format2"), ...){}. Is it worth translating the functions with an approximation that handles most of the use cases?Some testing that might be useful when putting together a PR:
asfimport commentedon Nov 18, 2021
Nicola Crane / @thisisnic:
@paleolimbot When you say 'approximated', how close or far is it?
asfimport commentedon Nov 18, 2021
Dewey Dunnington / @paleolimbot:
lubridate does it by ignoring all non-alpha/numeric characters so it'll happily parse something like "01:::::23::::45" just the same as "012345" or anywhere inbetween. I forget the exact incantations for strptime, but we could provide a list of formats we're willing to consider for each of them or preprocess with a regex to reduce the number of formats we need to consider (i.e.,
gsub("[^A-Za-z0-9.]", " ")). If that sounds close enough for you I'm happy to make a PR!asfimport commentedon Nov 18, 2021
Nicola Crane / @thisisnic:
Sounds like it'd be easy enough (famous last words!) to just strip out extra things with regex - so I'd say go ahead!
asfimport commentedon Feb 7, 2022
Dragoș Moldovan-Grünfeld / @dragosmg:
@paleolimbot I don't think we can rely on
coalesce()to iterate through the various formats supported for{}ymd(){}. It would need to rely on the assumption that the passedformatmatches the data or otherwise fail. Sadly, arrow works with a wrong format resulting in weird timestamps:Therefore, my early (and somewhat naive) conclusion would be that we cannot implement
arrow::ymd()binding as{}coalesce(strptime(x, format1), strptime(x, format2), ...){}. What do you think?asfimport commentedon Feb 7, 2022
Dragoș Moldovan-Grünfeld / @dragosmg:
We could to some processing to figure out how many characters we have (in the string to be parsed) in between the separators (or how many characters we have in total, in the cases where we have no separator) and only try with the suitable formats. i.e. in the example above not try to parse with
{}%Y{}, only{}%y{}.asfimport commentedon Feb 7, 2022
Dragoș Moldovan-Grünfeld / @dragosmg:
Another alternative would be for
strptimeto error when the selected format does not match the data (for example, attempting to parse"09-12-31"with{}"%Y-%m-%d"{}should error due to a mismatch in the length of the year). Then we could rely on this behaviour with{}coalesce{}.asfimport commentedon Feb 7, 2022
Dragoș Moldovan-Grünfeld / @dragosmg:
lubridateusesguess_formats()to identify the likely candidates. We could try something similar, where we have a list of supported formats (something similar to this), which we then narrow down to the most likely ones. Only then use something like{}coalesce(){}.