You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Feature request for support for struct and arry data types #3617
datafusion doesn't support all possible data types the arrow supports. What is the roadmap for supporting for structs, lists etc...? It would be good to support some pushdowns to the complex data to arrow.
thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ComputeError("concat requires input of at least one array"))', /home/kesav/.cargo/git/checkouts/arrow-datafusion-71ae82d9dec9a01c/8df5496/datafusion/common/src/scalar.rs:1383:18
Looking at datafusion/proto/src/physical_plan/to_proto.rs it does appear that this is not supported at present. Or perhaps I have made a mistake in my SQL?
The above mentioned error was when I ran the SQL from ballista and I've checked that ballista on the master branch is currently using datafusion version 18.0.0.
Hence, I just wrote two little tests, using the datafusion and the ballista context respectively.
SQL from the datafusion context works, whilst the one that uses the ballista context fails. Test code:
Yeah, it seems to work just fine for me in datafusion-cli. Thus I think we should close this ticket in datafusion. I am not sure what is going on with ballista.
No I take it back, what we should really do is probably start categorizing what works and what doesn't for these structureed types. Like array access via ["foo"] works, but not list access, for example.
Yeah, it's a good idea to document the features that already work for nested types as there are several cases that work already.
I'll take a look to see if I can find out where ballista is going wrong, although, of course, ultimately, the call ends in datafusion.
Looks like the protobuf serialisation is missing in ballista (for the GetIndexedFieldExpr), which is specific to ballista from what I can read. Hopefully an easy fix.
Activity
alamb commentedon Sep 28, 2022
You may find more information on #2326 -- it would be great to get some idea of what you are trying to do / what datafusion can't do for you today
kesavkolla commentedon Oct 6, 2022
I want to be able to specify so.e kind of path expression like select a.b.c for nested structs. For list types also some notation to access the index.
My data is a heavy nested and list structs. Currently I can't query them individual fields can't use nested columns in filters.
alamb commentedon Oct 6, 2022
Have you tried the
[]syntax?Like
struct_column["b"]["c"]for nested structs andlist_column[3]for list access? I was pleasantly surprised that it worked when I last tried itkesavkolla commentedon Oct 6, 2022
I tried for list list_column[0] it didn't work.
I get following exception:
ahmedriza commentedon Feb 15, 2023
If we take the Parquet provided by @kesavkolla, we have the following column,
textwhose Parquet schema is:and sample data:
I tried the following SQL to select one of the fields in the
struct:However, this resulted in the following error:
Looking at
datafusion/proto/src/physical_plan/to_proto.rsit does appear that this is not supported at present. Or perhaps I have made a mistake in my SQL?alamb commentedon Feb 15, 2023
Hi @ahmedriza -- I am not sure what your system is doing exactly, but that error appears to be related to protobuf serialization
I looked at
to_protoand it seems like it has the right codehttps://github.com/apache/arrow-datafusion/blob/d05647c65e14d865b854a845ae970797a6086e2c/datafusion/proto/src/logical_plan/to_proto.rs#L859C30-L867
Could you share the file you are using on this ticket so I can give it a try? Maybe we have fixed this in another version of DataFusion
ahmedriza commentedon Feb 15, 2023
@alamb Apologies, I should have been more clear. The Parquet file mentioned was in #2439. Attaching here as well:
part-00000-f6337bce-7fcd-4021-9f9d-040413ea83f8-c000.snappy.parquet.zip
The above mentioned error was when I ran the SQL from
ballistaand I've checked thatballistaon the master branch is currently usingdatafusionversion18.0.0.Hence, I just wrote two little tests, using the
datafusionand theballistacontext respectively.SQL from the
datafusioncontext works, whilst the one that uses theballistacontext fails. Test code:Output:
Output:
Please note that the error is coming from
datafusion/proto/src/physical_plan/to_proto.rsand not thelogical_plan.Here are the relevant parts of my
Cargo.toml.alamb commentedon Feb 16, 2023
Yeah, it seems to work just fine for me in datafusion-cli. Thus I think we should close this ticket in datafusion. I am not sure what is going on with ballista.
alamb commentedon Feb 16, 2023
No I take it back, what we should really do is probably start categorizing what works and what doesn't for these structureed types. Like array access via
["foo"]works, but not list access, for example.ahmedriza commentedon Feb 16, 2023
Yeah, it's a good idea to document the features that already work for nested types as there are several cases that work already.
I'll take a look to see if I can find out where
ballistais going wrong, although, of course, ultimately, the call ends indatafusion.Looks like the
protobufserialisation is missing inballista(for theGetIndexedFieldExpr), which is specific toballistafrom what I can read. Hopefully an easy fix.alamb commentedon Feb 16, 2023
Thanks @ahmedriza !
ahmedriza commentedon Feb 17, 2023
Raised #5324 that will fix the issue with
ballista.alamb commentedon Jun 23, 2023
#2326 is tracking such support
alamb commentedon Jun 19, 2024
I think we have significant support now for structs / lists so closing this down in favor of more specific asks / feature requests