You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I cannot able to downgrade arrow from version 2.0.0 to 1.0.1 since arrow::install_arrow() always installs the latest version.
Steps to reproduce:
devtools::install_version("arrow", version = "1.0.1")
arrow::install_arrow()
If I skip the arrow::install_arrow(), I am not able to use the gzip compression (WARNING: Arrow Gzip is not available, try using arrow::install_arrow())
Environment: R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 19.10
Neal Richardson / @nealrichardson:
Try setting the environment variable NOT_CRAN=true prior to install_version. That should give you a build with all of the compression libraries, so there's no need to arrow::install_arrow().
Vinicius Pinto:
I am debugging an error (IOError: Couldn't deserialize thrift: TProtocolException: Exceeded size limit) that appears when I use the latest versions of arrow (2.0) and readr(1.4). A tiny starvz data frame with only 4 rows becomes a parquet file with several MB (it seems a memory leak since the final size varies with the size of other starvz data frames) and this does not happen when using older versions.
Vinicius Pinto:
I prepared a "small case" (8M) that triggers this error. With smaller data frames, it works, but the parquet files are larger than expected.
Download link with the data and the commands to reproduce the issue: https://filesender.rnp.br/?s=download&token=cee89b89-3c75-49e6-b69f-db47cc2ab737
With this dataset, the total size of the parquet files increases from 8M with arrow 1.0.1 to 620M with arrow 2.0.0
Activity
asfimport commentedon Nov 16, 2020
Neal Richardson / @nealrichardson:
Try setting the environment variable
NOT_CRAN=trueprior toinstall_version. That should give you a build with all of the compression libraries, so there's no need toarrow::install_arrow().asfimport commentedon Nov 16, 2020
Neal Richardson / @nealrichardson:
See http://arrow.apache.org/docs/r/articles/install.html for details
asfimport commentedon Nov 16, 2020
Neal Richardson / @nealrichardson:
FTR that warning message is not coming from the
arrowpackage: https://github.com/schnorr/starvz/blob/87e1fbc8b152e720096fc3d9b9387528c9b617f2/R/write_functions.R#L24asfimport commentedon Nov 16, 2020
Neal Richardson / @nealrichardson:
Out of curiosity, why do you want to downgrade?
asfimport commentedon Nov 16, 2020
Vinicius Pinto:
Thanks @nealrichardson
I just add
Sys.setenv(NOT_CRAN = "true")before theinstall_versionand that works.asfimport commentedon Nov 16, 2020
Vinicius Pinto:
I am debugging an error (
IOError: Couldn't deserialize thrift: TProtocolException: Exceeded size limit)that appears when I use the latest versions of arrow (2.0) and readr(1.4). A tiny starvz data frame with only 4 rows becomes a parquet file with several MB (it seems a memory leak since the final size varies with the size of other starvz data frames) and this does not happen when using older versions.asfimport commentedon Nov 16, 2020
Neal Richardson / @nealrichardson:
Hmm. Can you share a tiny data frame that triggers this? Sounds like something that should be fixed.
asfimport commentedon Nov 17, 2020
Vinicius Pinto:
I prepared a "small case" (8M) that triggers this error. With smaller data frames, it works, but the parquet files are larger than expected.
Download link with the data and the commands to reproduce the issue: https://filesender.rnp.br/?s=download&token=cee89b89-3c75-49e6-b69f-db47cc2ab737
With this dataset, the total size of the parquet files increases from 8M with arrow 1.0.1 to 620M with arrow 2.0.0
asfimport commentedon Jan 4, 2021
Neal Richardson / @nealrichardson:
Issue resolved by pull request 9092
#9092