You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This issue is similar to #3804 but I couldn't find a corresponding JIRA issue.
I have a pandas dataframe (created from json file) where one column contains a python dictionary (type):
The type pyarrow detects with pa.DictionaryArray.from_pandas(df['type']).type is also struct<type_1: string, type_2: string>.
However, when building the pyarrow table with table = pa.Table.from_pandas(df, table_schema), an error always appears :
Traceback (most recent call last):
File "test_pyarrow.py", line 29, in table = pa.Table.from_pandas(df, table_schema)
File "pyarrow\table.pxi", line 1141, in pyarrow.lib.Table.from_pandas
File "C:\Users####\AppData\Local\Programs\Python\Python37\lib\site-packages\pyarrow\pandas_compat.py", line 443,
types
File "C:\Users####\AppData\Local\Programs\Python\Python37\lib\site-packages\pyarrow\pandas_compat.py", line 217,
df.columns, column_names, df_types
File "C:\Users####\AppData\Local\Programs\Python\Python37\lib\site-packages\pyarrow\pandas_compat.py", line 216,
) for col_name, sanitized_name, arrow_type in zip(
File "C:\Users####\AppData\Local\Programs\Python\Python37\lib\site-packages\pyarrow\pandas_compat.py", line 160,
logical_type = get_logical_type(arrow_type)
File "C:\Users####\AppData\Local\Programs\Python\Python37\lib\site-packages\pyarrow\pandas_compat.py", line 87, i
raise NotImplementedError(str(arrow_type))
NotImplementedError: struct<type_2: string, type_1: string>
Are nested dictionaries not implemented yet ? Or is there a way to declare a Dictionary in the schema (with DictionaryType) ?
Reacted by OlimSam, Nick Ireland, i4never and ryanzadera
Found a way to do it, so here's a quick summary of the code. I'll admit it looks quite like a hack but it still works :
import pandas as pd
import pyarrow as pa
# Data is just as in my previous comment
data = [ { "type": {"type_1" : "test_1", "type_2":"test_2"}, "id": "foo", "name": "Bar" }, { "type": {"type_1" : "test_1", "type_2":"test_2"}, "id": "foo_1", "name": "Bar_1" } ]
df = pd.DataFrame(data)
# Create a first table without the nested type
table = pa.Table.from_pandas(df[['id','name']])
# Get your nested pandas column and put it in a pyarrow column
column_to_add = pa.column('type', pa.array(df["type"]))
# Insert it at the position you want (mine was 0)
table = table.add_column(0, column_to_add)
print(table.to_pandas())
type
id
name
0
{'type_1': 'test_1', 'type_2': 'test_2'}
foo
bar
1
{'type_1': 'test_1', 'type_2': 'test_2'}
foo_1
Bar_1
And it keeps the nested structure in the "type" column. So it kind of works, but only kind of :
When getting the schema of that table and trying to re-use it, it doesn't work :
# Removed 3rd because it is the __index_level_0__ column and doesn't belong
schema_to_use = table.schema.remove(3)
pa_table_from_df = pa.Table.from_pandas(df, schema = schema_to_use)
Yields the same error as before : NotImplementedError: struct<type_1: string, type_2: string>
My goal was to save the table as parquet, and that didn't work either :
import pyarrow.parquet as pq
pq.write_table(table, 'test.parquet')
File "C:\Users\####\AppData\Local\Programs\Python\Python37\lib\site-packages\pyarro
writer.write_table(table, row_group_size=row_group_size)
File "C:\Users\####\AppData\Local\Programs\Python\Python37\lib\site-packages\pyarro
self.writer.write_table(table, row_group_size=row_group_size)
File "pyarrow_parquet.pyx", line 924, in pyarrow._parquet.ParquetWriter.write_table
File "pyarrow\error.pxi", line 81, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Nested column branch had multiple children
Sorry for the delay in answering this. As you noticed, an issue already exists for the Parquet issue.
As for the Pandas-to-Arrow conversion issue, I opened ARROW-5161.
Activity
MasterSalami commentedon Mar 28, 2019
Found a way to do it, so here's a quick summary of the code. I'll admit it looks quite like a hack but it still works :
And it keeps the nested structure in the "type" column. So it kind of works, but only kind of :
Yields the same error as before : NotImplementedError: struct<type_1: string, type_2: string>
Following https://jira.apache.org/jira/browse/ARROW-2587 and apache/parquet-cpp#462, it seems that saving to parquet with nested columns isn't implemented yet.
If I'm wrong, I'd love to be corrected on this 😅
Thanks in advance
pitrou commentedon Apr 11, 2019
Sorry for the delay in answering this. As you noticed, an issue already exists for the Parquet issue.
As for the Pandas-to-Arrow conversion issue, I opened ARROW-5161.
harikrishnan-dev commentedon Sep 27, 2019
I am still getting this error when I tried to save a nested json into a parquet format. I followed these steps
my json has few struct types
I ge this error
ArrowInvalid: Nested column branch had multiple childrenwesm commentedon Sep 27, 2019
Can you please open a JIRA issue with a reproducible example?