You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Python] Categorical boolean column saved as regular boolean in parquet #29017
When saving a pandas dataframe to parquet, if there is a categorical column where the categories are boolean, the column is saved as regular boolean.
This causes an issue because, when reading back the parquet file, I expect the column to still be categorical.
Reproducible example:
importpandasaspdimportpyarrow# Create dataframe with boolean column that is then converted to categoricaldf=pd.DataFrame({'a': [True, True, False, True, False]})
df['a'] =df['a'].astype('category')
# Convert to arrow Table and save to disktable=pyarrow.Table.from_pandas(df)
pyarrow.parquet.write_table(table, 'test.parquet')
# Reload data and convert back to pandastable_rel=pyarrow.parquet.read_table('test.parquet')
df_rel=table_rel.to_pandas()
The arrow table variable correctly converts the column to an arrow DICTIONARY type:
This in general seems a fairly reasonable behaviour to me, because in general binary data/text is where you get most saving due to the delta in size between the data and the indices.
On the other side, I don't think that "preserving same exact type" that was provided to a writer is an expectation that can be always satisfied or that it makes sense to enforce. In some cases enforcing dictionary encoding because the input was dictionary encoded might lead to bigger parquet files, or for some types for example that might not even be possible, think of CSV or JSON, not all types can be represented in those formats and thus the data you read back might have a different type
Weston Pace / @westonpace:
Judging by ARROW-3246 I believe the original intent was to support more types than just the binary types. Also, Arrow is supposed to be storing custom metadata (arrow:SCHEMA) in the parquet file with details on the original Arrow schema. This way, if we need to store the data more efficiently then we can do so while still restoring the original types upon deserialization. I checked and store_schema is set to true and it does properly decode and restore the original schema which does report the column "a" as dictionary<values=bool, indices=int8, ordered=0>. I'm not sure if some linkage is broken or if it only tries to restore the metadata on binary types.
However, I think the resolution for this issue is (although maybe not easy), the existing mechanism should be extended to handle more types.
Joris Van den Bossche / @jorisvandenbossche:
Directly reading dictionary encoded data in Parquet as dictionary typed arrays in Arrow is currently indeed only supported for BYTE_ARRAY storage (i.e. string, binary). The issue tracking the follow-up to expand this support to other data types is ARROW-6140.
The documentation of pq.read_table also somewhat mentions this:
This parameter allows to explicitly specify which columns to read as dictionary arrays. But it doesn't mention the default behaviour, which is to infer those columns from the stored arrow:SCHEMA.
And it is coded in https://github.com/apache/arrow/blob/14b75ee71d770ba86999e0e7a0e0b94629b91968/cpp/src/parquet/arrow/schema.cc#L410-L413 for which types this is currently supported to read as dictionary type:
In summary, I think the long term fix is ARROW-6140, and short term for this issue there are two options:
better document the current behaviour
cast afterwards to dictionary type, based on the ARROW:schema or pandas metadata (but this can be costly, after first materializing the full column, and not necessarily always wanted)
Alessandro Molina / @amol-:
I'm interested in taking a shot at ARROW-6140 unless it becomes urgent and someone more experience than me with the C++/Parquet side of things wants to take over. I'll take the ticket for the moment, anyone should feel free to take it over if there is extra urgency.
This still appears consistent with the earlier analysis in this thread: IsDictionaryReadSupported currently only allows binary/string dictionary reads in cpp/src/parquet/arrow/schema.cc.
Related note: the broader tracker for non-BYTE_ARRAY direct dictionary decoding, #22534 / ARROW-6140, is now closed as stale rather than fixed.
should this issue stay open as the Python-facing symptom, or should the non-BYTE_ARRAY dictionary decoding tracker be reopened / replaced with a fresh C++ Parquet issue? Happy to help keep a small reproducer or regression test attached once maintainers confirm the preferred direction.
Activity
asfimport commentedon Jul 16, 2021
Alessandro Molina / @amol-:
FYI, this seems to happen for numeric types. The same is true for
intanddoubleThe exception seem to be
unicodewhich preserves the dictionary form.In general, from my understanding, it seems arrow currently writes in dictionary form only the binary types (
arrow/cpp/src/parquet/column_writer.cc
Line 1008 in 14b75ee
This in general seems a fairly reasonable behaviour to me, because in general binary data/text is where you get most saving due to the delta in size between the data and the indices.
On the other side, I don't think that "preserving same exact type" that was provided to a writer is an expectation that can be always satisfied or that it makes sense to enforce. In some cases enforcing dictionary encoding because the input was dictionary encoded might lead to bigger parquet files, or for some types for example that might not even be possible, think of CSV or JSON, not all types can be represented in those formats and thus the data you read back might have a different type
asfimport commentedon Jul 16, 2021
Weston Pace / @westonpace:
Judging by ARROW-3246 I believe the original intent was to support more types than just the binary types. Also, Arrow is supposed to be storing custom metadata (arrow:SCHEMA) in the parquet file with details on the original Arrow schema. This way, if we need to store the data more efficiently then we can do so while still restoring the original types upon deserialization. I checked and store_schema is set to true and it does properly decode and restore the original schema which does report the column "a" as dictionary<values=bool, indices=int8, ordered=0>. I'm not sure if some linkage is broken or if it only tries to restore the metadata on binary types.
However, I think the resolution for this issue is (although maybe not easy), the existing mechanism should be extended to handle more types.
asfimport commentedon Jul 17, 2021
Joris Van den Bossche / @jorisvandenbossche:
Directly reading dictionary encoded data in Parquet as dictionary typed arrays in Arrow is currently indeed only supported for BYTE_ARRAY storage (i.e. string, binary). The issue tracking the follow-up to expand this support to other data types is ARROW-6140.
The documentation of
pq.read_tablealso somewhat mentions this:This parameter allows to explicitly specify which columns to read as dictionary arrays. But it doesn't mention the default behaviour, which is to infer those columns from the stored arrow:SCHEMA.
asfimport commentedon Jul 17, 2021
Joris Van den Bossche / @jorisvandenbossche:
This is handled in
ApplyOriginalStorageMetadataatarrow/cpp/src/parquet/arrow/schema.cc
Lines 879 to 889 in 14b75ee
And it is coded in
https://github.com/apache/arrow/blob/14b75ee71d770ba86999e0e7a0e0b94629b91968/cpp/src/parquet/arrow/schema.cc#L410-L413for which types this is currently supported to read as dictionary type:arrow/cpp/src/parquet/arrow/schema.cc
Lines 410 to 413 in 14b75ee
—
In summary, I think the long term fix is ARROW-6140, and short term for this issue there are two options:
asfimport commentedon Jul 17, 2021
Joris Van den Bossche / @jorisvandenbossche:
Some more explanation of the situation also in ARROW-11157
asfimport commentedon Jul 19, 2021
Alessandro Molina / @amol-:
I'm interested in taking a shot at ARROW-6140 unless it becomes urgent and someone more experience than me with the C++/Parquet side of things wants to take over. I'll take the ticket for the moment, anyone should feel free to take it over if there is extra urgency.
jingyi-zhao-01 commentedon Jul 13, 2026
I reproduced this on current Arrow main / local PyArrow dev build:
minimal reproducible code
26.0.0.dev3+gb03b4c5393.0.3Observed round-trip behavior:
This still appears consistent with the earlier analysis in this thread:
IsDictionaryReadSupportedcurrently only allows binary/string dictionary reads incpp/src/parquet/arrow/schema.cc.Related note: the broader tracker for non-
BYTE_ARRAYdirect dictionary decoding, #22534 / ARROW-6140, is now closed as stale rather than fixed.should this issue stay open as the Python-facing symptom, or should the non-
BYTE_ARRAYdictionary decoding tracker be reopened / replaced with a fresh C++ Parquet issue? Happy to help keep a small reproducer or regression test attached once maintainers confirm the preferred direction.