You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently reading parquet files generated by Hadoop (EMR) from S3 fails with "ValueError: >Found files in an intermediate directory" because of the _$folder$ empty files.
This was fixed in the pyarrow but R still has this issue.
The R side does not seem to have something similar to:
def _should_silently_exclude(self, file_name): return (file_name.endswith('.crc') or # Checksums file_name.endswith('_$folder$') or # HDFS directories in S3 file_name.startswith('.') or # Hidden files starting with . file_name.startswith('_') or # Hidden files starting with _ file_name in EXCLUDED_PARQUET_PATHS)
The code in pyarrow where this is mentioned looks to be in the legacy ParquetDataset code path, which predates the Dataset API we use in R. I believe this is being deprecated (right @jorisvandenbossche?). So it's not a drop-in fix for R.
There is a TODO to expose the FileSystemFactoryOptions in R, which we should do here. That struct has a selector_ignore_prefixes option, but since this _$folder$ is a suffix, it won't help. In fact, there are some misleading C++ tests (such as
) that include the _$folder$ string but not as a suffix like this. Perhaps we should add an ignore_suffixes option, or some other interface for filtering like this (cc @bkietz)
However, FileSystemFactoryOptions also has an exclude_invalid_files option, default false, and if true should successfully exclude _$folder$ files because they are empty. So exposing FileSystemFactoryOptions in R will allow you to flip this and that should do the trick.
The code in pyarrow where this is mentioned looks to be in the legacy ParquetDataset code path, which predates the Dataset API we use in R. I believe this is being deprecated (right Joris Van den Bossche?). So it's not a drop-in fix for R.
Yes, that's correct (so this is actually also an issue for python)
Perhaps we should add an ignore_suffixes option, or some other interface for filtering like this (cc Ben Kietzman)
I was thinking exactly the same. Based on the python snippet from the legacy code, it is clear that it's ignoring some files both based on prefixes and suffixes.
It might be possible to have some more advanced / smarter (callback based?) filename filter option, but I suppose that a simpler prefix+suffix ignore options will cover almost all use cases?
Activity
asfimport commentedon Jan 31, 2022
Dewey Dunnington / @paleolimbot:
Thank you for reporting this!
Where this lives in Python:
arrow/python/pyarrow/parquet.py
Lines 1181 to 1186 in ad073b7
We don' have something similar to this approach in R, but it looks like these are exposed in the C++ API as well:
arrow/cpp/src/arrow/dataset/discovery.h
Lines 146 to 195 in f8661e0
...which we could wire up to R here:
arrow/r/src/dataset.cpp
Lines 171 to 172 in 03219e2
and here:
arrow/r/R/dataset.R
Line 181 in 03219e2
asfimport commentedon May 14, 2022
Neal Richardson / @nealrichardson:
A couple of observations as I start to look into this:
selector_ignore_prefixesoption, but since this _$folder$ is a suffix, it won't help. In fact, there are some misleading C++ tests (such asarrow/cpp/src/arrow/dataset/discovery_test.cc
Lines 225 to 239 in 2e8b836
exclude_invalid_filesoption, default false, and if true should successfully exclude _$folder$ files because they are empty. So exposing FileSystemFactoryOptions in R will allow you to flip this and that should do the trick.asfimport commentedon May 17, 2022
Joris Van den Bossche / @jorisvandenbossche:
Yes, that's correct (so this is actually also an issue for python)
I was thinking exactly the same. Based on the python snippet from the legacy code, it is clear that it's ignoring some files both based on prefixes and suffixes.
It might be possible to have some more advanced / smarter (callback based?) filename filter option, but I suppose that a simpler prefix+suffix ignore options will cover almost all use cases?
asfimport commentedon May 19, 2022
Neal Richardson / @nealrichardson:
Issue resolved by pull request 13171
#13171