You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a parquet file with schema of nested array. There are several levels where type is array but data is empty. When use this data with datafusion I am getting this error:
ArrowError(ComputeError("concat requires input of at least one array"))
Even if use projection with only non nested columns then also I am getting the above error. Can you someone share thoughts on how can I get away with this error?
Yup, I can confirm with apache/arrow-rs#1700 it is possible for the parquet reader to handle the file. Unfortunately work remains to support this on DataFusion's side, e.g. #2453. I fully intend to get this working, there just be many bugs along the way 😅
Activity
andygrove commentedon May 4, 2022
@kesavkolla Is it possible that you could provide a minimal repro parquet file that we can use for testing?
kesavkolla commentedon May 4, 2022
@andygrove
I had to create a zip file from the compressed parquet because github is not letting to updload files otherwise.
Just unzip the file and try to load it via datafusion then we get the above error.
Here is my code snippet
part-00000-f6337bce-7fcd-4021-9f9d-040413ea83f8-c000.snappy.parquet.zip
alamb commentedon May 13, 2022
I believe this will be fixed in arrow 15 (in a few weeks) due to @tustvold 's work in apache/arrow-rs#1682
tustvold commentedon May 13, 2022
Yup, I can confirm with apache/arrow-rs#1700 it is possible for the parquet reader to handle the file. Unfortunately work remains to support this on DataFusion's side, e.g. #2453. I fully intend to get this working, there just be many bugs along the way 😅
kesavkolla commentedon May 13, 2022
Looking forward to do early testing
kesavkolla commentedon May 13, 2022
I wish this is included in arrow 14 itself. :-)
HaoYang670 commentedon Sep 21, 2022
Could we close this issue?
alamb commentedon Sep 21, 2022
I just tried this file with @kesavkolla 's example using
datafusion-cliand it does appear to run without error now:so closing as complete