Skip to content

[Python] Categorical boolean column saved as regular boolean in parquet #29017

Open

Description

@asfimport

When saving a pandas dataframe to parquet, if there is a categorical column where the categories are boolean, the column is saved as regular boolean.

This causes an issue because, when reading back the parquet file, I expect the column to still be categorical.

 
Reproducible example:

import pandas as pd
import pyarrow

# Create dataframe with boolean column that is then converted to categorical
df = pd.DataFrame({'a': [True, True, False, True, False]})
df['a'] = df['a'].astype('category')

# Convert to arrow Table and save to disk
table = pyarrow.Table.from_pandas(df)
pyarrow.parquet.write_table(table, 'test.parquet')

# Reload data and convert back to pandas
table_rel = pyarrow.parquet.read_table('test.parquet')
df_rel = table_rel.to_pandas()

The arrow table variable correctly converts the column to an arrow DICTIONARY type:


>>> df['a']
0     True
1     True
2    False
3     True
4    False
Name: a, dtype: category
Categories (2, object): [False, True]
>>>
>>> table
pyarrow.Table
a: dictionary<values=bool, indices=int8, ordered=0>

However, the reloaded column is now a regular boolean:


>>> table_rel
pyarrow.Table
a: bool
>>>
>>> df_rel['a']
0     True
1     True
2    False
3     True
4    False
Name: a, dtype: bool

I would have expected the column to be read back as categorical.

Reporter: Joao Moreira

Related issues:

Note: This issue was originally created as ARROW-13342. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Jul 16, 2021

    @asfimport
    Author

    Alessandro Molina / @amol-:
    FYI, this seems to happen for numeric types. The same is true for int and double

    import pyarrow as pa
    import pyarrow.parquet
    
    # Convert to arrow Table and save to disk
    table = pa.Table.from_arrays([
        pa.DictionaryArray.from_arrays(indices=pa.array([0, 1, 1, 0, 1, 1, 0], type=pa.int8()), 
                                       dictionary=[10000000000000000, 20000000000000000])
    ], names=["data"])
    print(table)
    pa.parquet.write_table(table, 'test.parquet')
    
    # Reload data and convert back to pandas
    table_rel = pa.parquet.read_table('test.parquet')
    print(table_rel)

    The exception seem to be unicode which preserves the dictionary form.

    In general, from my understanding, it seems arrow currently writes in dictionary form only the binary types (

    return ::arrow::is_base_binary_like(dict_type.value_type()->id());
    )

    This in general seems a fairly reasonable behaviour to me, because in general binary data/text is where you get most saving due to the delta in size between the data and the indices.

    On the other side, I don't think that "preserving same exact type" that was provided to a writer is an expectation that can be always satisfied or that it makes sense to enforce. In some cases enforcing dictionary encoding because the input was dictionary encoded might lead to bigger parquet files, or for some types for example that might not even be possible, think of CSV or JSON, not all types can be represented in those formats and thus the data you read back might have a different type

  2. asfimport commented on Jul 16, 2021

    @asfimport
    Author

    Weston Pace / @westonpace:
    Judging by ARROW-3246 I believe the original intent was to support more types than just the binary types.  Also, Arrow is supposed to be storing custom metadata (arrow:SCHEMA) in the parquet file with details on the original Arrow schema.  This way, if we need to store the data more efficiently then we can do so while still restoring the original types upon deserialization.  I checked and store_schema is set to true and it does properly decode and restore the original schema which does report the column "a" as dictionary<values=bool, indices=int8, ordered=0>.  I'm not sure if some linkage is broken or if it only tries to restore the metadata on binary types.

     

    However, I think the resolution for this issue is (although maybe not easy), the existing mechanism should be extended to handle more types.

  3. asfimport commented on Jul 17, 2021

    @asfimport
    Author

    Joris Van den Bossche / @jorisvandenbossche:
    Directly reading dictionary encoded data in Parquet as dictionary typed arrays in Arrow is currently indeed only supported for BYTE_ARRAY storage (i.e. string, binary). The issue tracking the follow-up to expand this support to other data types is ARROW-6140.

    The documentation of pq.read_table also somewhat mentions this:

    read_dictionary : list, default None
        List of names or column paths (for nested types) to read directly
        as DictionaryArray. Only supported for BYTE_ARRAY storage. 
        ...

    This parameter allows to explicitly specify which columns to read as dictionary arrays. But it doesn't mention the default behaviour, which is to infer those columns from the stored arrow:SCHEMA.

  4. asfimport commented on Jul 17, 2021

    @asfimport
    Author

    Joris Van den Bossche / @jorisvandenbossche:

    But it doesn't mention the default behaviour, which is to infer those columns from the stored arrow:SCHEMA.

    This is handled in ApplyOriginalStorageMetadata at

    if (origin_type->id() == ::arrow::Type::DICTIONARY &&
    inferred_type->id() != ::arrow::Type::DICTIONARY &&
    IsDictionaryReadSupported(*inferred_type)) {
    // Direct dictionary reads are only suppored for a couple primitive types,
    // so no need to recurse on value types.
    const auto& dict_origin_type =
    checked_cast<const ::arrow::DictionaryType&>(*origin_type);
    inferred->field = inferred->field->WithType(
    ::arrow::dictionary(::arrow::int32(), inferred_type, dict_origin_type.ordered()));
    modified = true;
    }

    And it is coded in https://github.com/apache/arrow/blob/14b75ee71d770ba86999e0e7a0e0b94629b91968/cpp/src/parquet/arrow/schema.cc#L410-L413 for which types this is currently supported to read as dictionary type:

    bool IsDictionaryReadSupported(const ArrowType& type) {
    // Only supported currently for BYTE_ARRAY types
    return type.id() == ::arrow::Type::BINARY || type.id() == ::arrow::Type::STRING;
    }

    —

    In summary, I think the long term fix is ARROW-6140, and short term for this issue there are two options:

    • better document the current behaviour
    • cast afterwards to dictionary type, based on the ARROW:schema or pandas metadata (but this can be costly, after first materializing the full column, and not necessarily always wanted)
  5. asfimport commented on Jul 17, 2021

    @asfimport
    Author

    Joris Van den Bossche / @jorisvandenbossche:
    Some more explanation of the situation also in ARROW-11157

  6. asfimport commented on Jul 19, 2021

    @asfimport
    Author

    Alessandro Molina / @amol-:
    I'm interested in taking a shot at ARROW-6140 unless it becomes urgent and someone more experience than me with the C++/Parquet side of things wants to take over. I'll take the ticket for the moment, anyone should feel free to take it over if there is extra urgency.

  7. jingyi-zhao-01 commented on Jul 13, 2026

    @jingyi-zhao-01

    I reproduced this on current Arrow main / local PyArrow dev build:

    minimal reproducible code

    from pathlib import Path
    import tempfile
    
    try:
        import pandas as pd
        import pyarrow as pa
        import pyarrow.parquet as pq
    except ModuleNotFoundError as exc:
        raise SystemExit(
            f"Missing dependency: {exc.name}\n"
            "Run: python -m pip install pyarrow pandas"
        ) from exc
    
    
    def roundtrip(name, values):
        df = pd.DataFrame({"a": values})
        df["a"] = df["a"].astype("category")
    
        table = pa.table(df)
        path = Path(tempfile.mkdtemp(prefix="arrow-29017-")) / f"{name}.parquet"
        pq.write_table(table, path)
    
        read_table = pq.read_table(path)
        read_df = read_table.to_pandas()
    
        print(f"\n{name}")
        print(f"  parquet file: {path}")
        print(f"  original pandas dtype: {df['a'].dtype}")
        print(f"  written arrow type:    {table.schema.field('a').type}")
        print(f"  read arrow type:       {read_table.schema.field('a').type}")
        print(f"  read pandas dtype:     {read_df['a'].dtype}")
        print(f"  pandas category kept:  {read_df['a'].dtype.name == 'category'}")
    
    
    def main():
        print(f"pyarrow {pa.__version__}")
        print(f"pandas  {pd.__version__}")
    
        roundtrip("bool_category", [True, True, False, True, False])
        roundtrip("int_category", [1, 1, 2, 1, 2])
        roundtrip("string_category", ["a", "a", "b", "a", "b"])
    
    
    if __name__ == "__main__":
        main()
    
    • pyarrow 26.0.0.dev3+gb03b4c539
    • pandas 3.0.3

    Observed round-trip behavior:

    bool_category
      written arrow type:    dictionary<values=bool, indices=int8, ordered=0>
      read arrow type:       bool
      read pandas dtype:     bool
      pandas category kept:  False
    
    int_category
      written arrow type:    dictionary<values=int64, indices=int8, ordered=0>
      read arrow type:       int64
      read pandas dtype:     int64
      pandas category kept:  False
    
    string_category
      written arrow type:    dictionary<values=large_string, indices=int8, ordered=0>
      read arrow type:       dictionary<values=string, indices=int8, ordered=0>
      read pandas dtype:     category
      pandas category kept:  True
    

    This still appears consistent with the earlier analysis in this thread: IsDictionaryReadSupported currently only allows binary/string dictionary reads in cpp/src/parquet/arrow/schema.cc.

    Related note: the broader tracker for non-BYTE_ARRAY direct dictionary decoding, #22534 / ARROW-6140, is now closed as stale rather than fixed.

    should this issue stay open as the Python-facing symptom, or should the non-BYTE_ARRAY dictionary decoding tracker be reopened / replaced with a fresh C++ Parquet issue? Happy to help keep a small reproducer or regression test attached once maintainers confirm the preferred direction.

  8. added
    Status: needs championHigh impact issues which aren't being worked on but require a volunteer to move the task forward.
    on Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions