Skip to content

[R] Proactively remove "problems" attributes #26580

Closed

Description

@asfimport

I cannot able to downgrade arrow from version 2.0.0 to 1.0.1 since arrow::install_arrow() always installs the latest version.

Steps to reproduce:

 

devtools::install_version("arrow", version = "1.0.1")
arrow::install_arrow()

 

If I skip the arrow::install_arrow(), I am not able to use the gzip compression (WARNING: Arrow Gzip is not available, try using arrow::install_arrow())

 

Environment: R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 19.10

Reporter: Vinicius Pinto
Assignee: Jonathan Keane / @jonkeane

Original Issue Attachments:

PRs and other links:

Note: This issue was originally created as ARROW-10624. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Nov 16, 2020

    @asfimport
    Author

    Neal Richardson / @nealrichardson:
    Try setting the environment variable NOT_CRAN=true prior to install_version. That should give you a build with all of the compression libraries, so there's no need to arrow::install_arrow().

  2. asfimport commented on Nov 16, 2020

    @asfimport
    Author
  3. asfimport commented on Nov 16, 2020

    @asfimport
    Author
  4. asfimport commented on Nov 16, 2020

    @asfimport
    Author

    Neal Richardson / @nealrichardson:
    Out of curiosity, why do you want to downgrade?

  5. asfimport commented on Nov 16, 2020

    @asfimport
    Author

    Vinicius Pinto:
    Thanks @nealrichardson
    I just add Sys.setenv(NOT_CRAN = "true") before the install_version and that works.

  6. asfimport commented on Nov 16, 2020

    @asfimport
    Author

    Vinicius Pinto:
    I am debugging an error (IOError: Couldn't deserialize thrift: TProtocolException: Exceeded size limit) that appears when I use the latest versions of arrow (2.0) and readr(1.4). A tiny starvz data frame with only 4 rows becomes a parquet file with several MB (it seems a memory leak since the final size varies with the size of other starvz data frames) and this does not happen when using older versions.

  7. asfimport commented on Nov 16, 2020

    @asfimport
    Author

    Neal Richardson / @nealrichardson:
    Hmm. Can you share a tiny data frame that triggers this? Sounds like something that should be fixed.

  8. asfimport commented on Nov 17, 2020

    @asfimport
    Author

    Vinicius Pinto:
    I prepared a "small case" (8M) that triggers this error. With smaller data frames, it works, but the parquet files are larger than expected.
    Download link with the data and the commands to reproduce the issue: https://filesender.rnp.br/?s=download&token=cee89b89-3c75-49e6-b69f-db47cc2ab737
    With this dataset, the total size of the parquet files increases from 8M with arrow 1.0.1 to 620M with arrow 2.0.0

     

     

  9. asfimport commented on Jan 4, 2021

    @asfimport
    Author

    Neal Richardson / @nealrichardson:
    Issue resolved by pull request 9092
    #9092

  10. added this to the 3.0.0 milestone on Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions