You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[R] When write parquet as data.frame we lost information about class. #44524
# Convert the data.frame to an Arrow Table
iris_table <- arrow::Table$create(iris)
# Retrieve existing metadata (if any)
existing_metadata <- iris_table$schema$metadata
# Add or update the 'class' metadata to indicate it's a data.frame
new_metadata <- existing_metadata
new_metadata$r$attributes$class <- c("data.frame", "custom.data.frame")
# Update the schema with the new metadata
iris_table <- iris_table$ReplaceSchemaMetadata(new_metadata)
arrow::write_parquet(iris_table, "iris.parquet")
Bug description:
When we remove data.frame class attribute, we read parquet by default as tibble. In my opinion it's not expected behavior as when we write data.frame we should read data.frame.
What was the reason for remove the class if it's just data.frame?
Hi @galachad, the behaviour you describe is not a bug but an intended feature. There's more on reasoning in #35173 and #34825. Would you mind expanding a bit more on the problems this is causing for you?
We have some legacy code that isn't compatible with tibble. When the data.frame attributes are removed, parquet files are saved without a class (starting from arrow version 13+). In the absence of the class attribute, Arrow reads the file as a tibble by default.
While I understand the intent to remove unnecessary attributes, the assumption that data.frame is the default format is incorrect. The default type for reading parquet files is actually tibble.
If the goal is to remove attributes for the default table type in R, we should remove the class only when it is c("tbl_df", "tbl", "data.frame"), not just 'data.frame'.
As I mention in the description, to avoid class modification, extra workaround is required now. Unfortunately, the tibble and data.frame are not 100% compatible.
The example that the assumption was made is probably incorrect:
It's not a bug - it was a deliberate design decision to have a default type of tibble - the discussion is in those linked PRs, though is spread a bit between multiple ones. Do you still have the same problems if you were to call as.data.frame() on the loaded object after calling read_parquet()?
I'm just pointing that if arrow default type is tibble then we should remove class attribute only when object is tibble.
As I mention before, when we save data.frame into parquet, then read_parquet returns tibble.
It could be changed to tibble - remove the class attribute when the object is just tibble.
# remove the class if it's just tibble
if (identical(x$attributes$class, c("tbl_df", "tbl", "data.frame"))) {
x$attributes <- x$attributes[names(x$attributes) != "class"]
if (is_empty(x$attributes)) {
x <- x[names(x) != "attributes"]
}
}
You can always convert tibble to data.frame, but I'm just pointing that current behavior is not consistence.
Anyway, if you are not agreed that removing class information from not default type object is a bug, then we should close the issue.
Activity
thisisnic commentedon Oct 28, 2024
Hi @galachad, the behaviour you describe is not a bug but an intended feature. There's more on reasoning in #35173 and #34825. Would you mind expanding a bit more on the problems this is causing for you?
galachad commentedon Oct 28, 2024
We have some legacy code that isn't compatible with
tibble. When thedata.frameattributes are removed, parquet files are saved without a class (starting from arrow version 13+). In the absence of theclassattribute, Arrow reads the file as atibbleby default.While I understand the intent to remove unnecessary attributes, the assumption that
data.frameis the default format is incorrect. The default type for reading parquet files is actuallytibble.If the goal is to remove attributes for the default table type in R, we should remove the class only when it is
c("tbl_df", "tbl", "data.frame"), not just'data.frame'.As I mention in the description, to avoid class modification, extra workaround is required now. Unfortunately, the
tibbleanddata.frameare not 100% compatible.The example that the assumption was made is probably incorrect:
In this case,
mtcarsis adata.frameand should be represented asdata.frame.In my opinion, this bug is tiny and can be resolved just by remove
remove the class if it's just data.framesection.arrow/r/R/metadata.R
Lines 25 to 31 in 7ef5437
thisisnic commentedon Oct 28, 2024
It's not a bug - it was a deliberate design decision to have a default type of tibble - the discussion is in those linked PRs, though is spread a bit between multiple ones. Do you still have the same problems if you were to call
as.data.frame()on the loaded object after callingread_parquet()?galachad commentedon Oct 29, 2024
I'm just pointing that if arrow default type is
tibblethen we should remove class attribute only when object istibble.As I mention before, when we save
data.frameinto parquet, thenread_parquetreturnstibble.It could be changed to tibble - remove the class attribute when the object is just tibble.
You can always convert
tibbletodata.frame, but I'm just pointing that current behavior is not consistence.Anyway, if you are not agreed that removing class information from not default type object is a bug, then we should close the issue.