Skip to content

[R] SF columns in datasets with filters #28305

Closed

Description

@asfimport

First reported at https://issues.apache.org/jira/browse/ARROW-10386?focusedCommentId=17331668&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17331668

OK, I actually have recreated a similar issue. In the following code, I create an sf object and write it as a dataset to parquet files. I then call open_dataset() on the files.

If I collect() the dataset I get back an sf object, no problem.

But if I first filter() the dataset then collect() I get an error.

library(sf)
library(arrow)
library(dplyr)

n <- 10000

fake <- tibble(
    ID=seq(n),
    Date=sample(seq(as.Date('2019-01-01'), as.Date('2021-04-01'), by=1), size=n, replace=TRUE),
    x=runif(n=n, min=-170, max=170),
    y=runif(n=n, min=-60, max=70),
    text1=sample(x=state.name, size=n, replace=TRUE),
    text2=sample(x=state.name, size=n, replace=TRUE),
    text3=sample(x=state.division, size=n, replace=TRUE),
    text4=sample(x=state.region, size=n, replace=TRUE),
    text5=sample(x=state.abb, size=n, replace=TRUE),
    num1=sample(x=state.center$x, size=n, replace=TRUE),
    num2=sample(x=state.center$y, size=n, replace=TRUE),
    num3=sample(x=state.area, size=n, replace=TRUE),
    Rand1=rnorm(n=n),
    Rand2=rnorm(n=n, mean=100, sd=3),
    Rand3=rbinom(n=n, size=10, prob=0.4)
)

# make it into an sf object
spat <- fake %>% 
    st_as_sf(coords=c('x', 'y'), remove=FALSE, crs = 4326)

class(spat)
class(spat$geometry)

# create new columns for partitioning and write to disk
spat %>% 
    mutate(Year=lubridate::year(Date), Month=lubridate::month(Date)) %>% 
    group_by(Year, Month) %>% 
    write_dataset('data/splits/', format='parquet')

spat_in <- open_dataset('data/splits/')

class(spat_in)

# it's an sf as expected
spat_in %>% collect() %>% class()
spat_in %>% collect() %>% pull(geometry) %>% class()

# it even plots
leaflet::leaflet() %>% 
    leaflet::addTiles() %>% 
    leafgl::addGlPoints(data=spat_in %>% collect())

# but if we filter first
spat_in %>% 
    filter(Year == 2020 & Month == 2) %>% 
    collect()

# we get this error
Error in st_geometry.sf(x) : 
  attr(obj, "sf_column") does not point to a geometry column.
Did you rename it, without setting st_geometry(obj) <- "newname"?
In addition: Warning message:
Invalid metadata$r 

Reporter: Jonathan Keane / @jonkeane
Assignee: Jonathan Keane / @jonkeane

Related issues:

PRs and other links:

Note: This issue was originally created as ARROW-12542. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Apr 26, 2021

    @asfimport
    Author

    Jonathan Keane / @jonkeane:
    Ok I'm starting to see what's going on here: When collecting the whole dataset everything is fine. But when we filter, we have the filtered data but try to apply the whole list of metdata to that column, but there's a length mismatch, so the metadata application stops (that's what that warning is that you're seeing) and thus the SF column isn't getting reconstructed correctly.

    This is the branch where that is being called, but we might want/need to filter the metadata higher https://github.com/apache/arrow/blob/master/r/R/metadata.R#L62-L65 I'm looking into this to see if we can fix it.

  2. asfimport commented on Apr 26, 2021

    @asfimport
    Author

    Jonathan Keane / @jonkeane:
    Oh, and interestingly, when writing the dataset it looks like when partitioning we're actually writing all of the metadata out to each partition (instead of partitioning it so that each partition only has the metadata for the rows in its partition)

    > df <- tibble::tibble(
    +   part = 1:10,
    +   a = rep(1:2, 5),
    +   x = rep(list(structure(1, foo = "bar"), structure(2, baz = "qux")), 5)
    + )
    > 
    > # write the dataset, and open it
    > tmp <- tempfile()
    > write_dataset(df, tmp, partitioning = "part", format = "parquet")
    > ds <- open_dataset(tmp)
    > 
    > read_parquet(file.path(tmp, "part=1/part-0.parquet"))
    # A tibble: 1 x 2
          a              x
      <int> <list<double>>
    1     1            [1]
    Warning message:
    Invalid metadata$r 
    > 
    > tab <- read_parquet(file.path(tmp, "part=1/part-0.parquet"), as_data_frame = FALSE)
    > 
    > tab$metadata$r
     'arrow_r_metadata' chr "A\n3\n262148\n197888\n5\nUTF-8\n531\n2\n531\n0\n1026\n1\n262153\n5\nnames\n16\n0\n254\n531\n3\n254\n254\n531\n2"| __truncated__
    List of 2
     $ attributes: Named list()
     $ columns   :List of 3
      ..$ part: NULL
      ..$ a   : NULL
      ..$ x   :List of 2
      .. ..$ attributes: NULL
      .. ..$ columns   :List of 10
      .. .. ..$ :List of 2
      .. .. .. ..$ attributes:List of 1
      .. .. .. .. ..$ foo: chr "bar"
      .. .. .. ..$ columns   : NULL
      .. .. ..$ :List of 2
      [...]

    Solving that alone (probably) won't be enough to resolve the filter() issue as well, but will be part of the solution

  3. asfimport commented on Apr 30, 2021

    @asfimport
    Author

    Neal Richardson / @nealrichardson:
    If we're storing per-row metadata in the schema, that's not going to behave well. Perhaps we need to rethink how we store this in a way that round-trips better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions