You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Run this function to write parquet files in this example please
write_chunk_data <- function(data_path, output_dir, chunk_size = 1000000) {
#If the output_dir do not exist, it is created
if (!fs::dir_exists(output_dir)) fs::dir_create(output_dir)
#It gets the name of the file
data_name <- fs::path_ext_remove(fs::path_file(data_path))
#It sets the chunk_num to 0
chunk_num <- 0
#Read the file using vroom
data_chunk <- vroom::vroom(data_path)
#It gets the variable names
data_names <- names(data_chunk)
#It gets the number of rows
rows<-nrow(data_chunk)
#The following loop creates a parquet file for every [chunk_size] rows
repeat{
#It checks if we are over the max rows
if(rows>(chunk_num+1)*chunk_size)
#This is to recover some memory and space in the disk
rm(data_chunk)
tmp_file <- tempdir()
files <- list.files(tmp_file, full.names = T, pattern = "^vroom")
file.remove(files)
}
Dragoș Moldovan-Grünfeld / @dragosmg:
Hey. Thanks for submitting the issue. I reorganised your small script a bit (see below) but it seems to be working for me. I tried to keep it as close to yours as possible. I changed the paths to point to my desktop. Also, when creating df you referenced 2 objects ({}tabla1_arrow{} and {}tabla2_arrow{}) which don't exists so I assumed you were referring to df1_arrow and {}df2_arrow{}.
I tried both with the latest CRAN version (‘6.0.1’) and the development version (‘6.0.1.9000’) and it works with both.
A small note: I don't think purrr::walk() is needed there, you can simply get away with calling your function with the desired inputs.
library(dplyr)
library(purrr)
library(arrow)
library(fs)
library(vroom)
library(glue)
data(iris)
temp_csv_file<- tempfile(fileext=".csv")
temp_parquet_dir<-fs::dir_create(tempdir(), "/parquet")
write.csv(iris, temp_csv_file) # write csv file# write parket file with write_chunk_data function (below)# Run this function to write parquet files in this example pleasewrite_chunk_data<-function(data_path, output_dir, chunk_size=1000000) {
#If the output_dir do not exist, it is createdif (!fs::dir_exists(output_dir)) fs::dir_create(output_dir)
#It gets the name of the filedata_name<-fs::path_ext_remove(fs::path_file(data_path))
#It sets the chunk_num to 0chunk_num<-0#Read the file using vroomdata_chunk<-vroom::vroom(data_path)
#It gets the variable namesdata_names<- names(data_chunk)
#It gets the number of rowsrows<-nrow(data_chunk)
#The following loop creates a parquet file for every [chunk_size] rowsrepeat{
#It checks if we are over the max rowsif(rows>(chunk_num+1)*chunk_size){
arrow::write_parquet(data_chunk[(chunk_num*chunk_size+1):((chunk_num+1)*chunk_size),],
fs::path(output_dir, glue::glue("{data_name}-{chunk_num}.parquet")))
}else{
arrow::write_parquet(data_chunk[(chunk_num*chunk_size+1):rows,],
fs::path(output_dir, glue::glue("{data_name}-{chunk_num}.parquet")))
break
}
chunk_num<-chunk_num+1
}
#This is to recover some memory and space in the disk
rm(data_chunk)
tmp_file<- tempdir()
files<- list.files(tmp_file, full.names=T, pattern="^vroom")
file.remove(files)
}
walk(temp_csv_file,
write_chunk_data, temp_parquet_dir, chunk_size=50)
iris_arrow<- open_dataset(temp_parquet_dir)
df1_arrow<-iris_arrow %>%
select(`...1`, Sepal.Length, Sepal.Width, Petal.Length)
df2_arrow<-iris_arrow %>%
select(`...1`, Petal.Width, Species,)
df<-df1_arrow %>%
inner_join(df2_arrow, by="...1") %>%
group_by(Species) %>%
summarise(prom= mean(Sepal.Length)) %>%
collect()
print(df)
Output:
# A tibble: 3 × 2Speciesprom<chr><dbl>1setosa5.012virginica6.593versicolor5.94
Dewey Dunnington / @paleolimbot:
I also tried to replicate this using Dragos' reprex and couldn't (on MacOS, arrow C++ and R from the master branch). I wonder if this is a duplicate of ARROW-14908?
Dragoș Moldovan-Grünfeld / @dragosmg:
It could be. I was also testing on macOS. [~Zea] would you be able to provide the output of utils::sessionInfo() or {}devtools::session_info(){}? My devtools one is below:
Nicola Crane / @thisisnic:
I've tried to replicate this (Dragos' reformulation) on Ubuntu 20.04 with R 4.1.2 and Arrow 6.0.1 but had no luck - code ran fine.
Activity
asfimport commentedon Jan 27, 2022
Dragoș Moldovan-Grünfeld / @dragosmg:
Hey. Thanks for submitting the issue. I reorganised your small script a bit (see below) but it seems to be working for me. I tried to keep it as close to yours as possible. I changed the paths to point to my desktop. Also, when creating
dfyou referenced 2 objects ({}tabla1_arrow{} and{}tabla2_arrow{}) which don't exists so I assumed you were referring todf1_arrowand{}df2_arrow{}.I tried both with the latest CRAN version (‘6.0.1’) and the development version (‘6.0.1.9000’) and it works with both.
A small note: I don't think
purrr::walk()is needed there, you can simply get away with calling your function with the desired inputs.Output:
asfimport commentedon Jan 31, 2022
Dewey Dunnington / @paleolimbot:
I also tried to replicate this using Dragos' reprex and couldn't (on MacOS, arrow C++ and R from the master branch). I wonder if this is a duplicate of ARROW-14908?
asfimport commentedon Feb 1, 2022
Dragoș Moldovan-Grünfeld / @dragosmg:
It could be. I was also testing on macOS.
[~Zea]would you be able to provide the output ofutils::sessionInfo()or{}devtools::session_info(){}? Mydevtoolsone is below:asfimport commentedon Feb 1, 2022
Nicola Crane / @thisisnic:
I've tried to replicate this (Dragos' reformulation) on Ubuntu 20.04 with R 4.1.2 and Arrow 6.0.1 but had no luck - code ran fine.