You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support writing lists in the arrow csv writer #4502
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Add support to the arrow CSV writer for writing lists.
Currently an sqllogictest of the format:
query T
SELECT array_agg(c13) FROM (SELECT * FROM aggregate_test_100 ORDER BY c13 LIMIT 2) test
----
[0VVIHzxWtNOFLtnhjHEKjXaJOSLJfm0keZ5G8BffGwgF2RwQD59TFzMStxCB]
will fail with thread 'main' panicked at 'called Result::unwrap() on an Err value: CsvError("CSV Writer does not support List(Field { name: \"item\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) data type")', datafusion/core/tests/sqllogictests/src/main.rs:134:33
This would be an upstream issue in arrow-rs if we wish to add support for this, and I would be happy to review a PR adding support for it.
That being said, I'm not sure how to encode nested data in CSV? I had understood the format to be strictly tabular? The docs for arrow python would appear to suggest only primitive types are supported although I haven't tested this?
Activity
tustvold commentedon Dec 7, 2022
This would be an upstream issue in arrow-rs if we wish to add support for this, and I would be happy to review a PR adding support for it.
That being said, I'm not sure how to encode nested data in CSV? I had understood the format to be strictly tabular? The docs for arrow python would appear to suggest only primitive types are supported although I haven't tested this?
FWIW arrow-rs supports reading/writing nested JSON
melgenek commentedon Jan 24, 2023
Sqllogictest doesn't use the CSV writer anymore #4578.
The test from this issue passes successfully https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/tests/sqllogictests/test_files/aggregate.slt#L953-L957.
@xudong963 @alamb @tustvold I guess, this ticket can be closed.