You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Python] CategoricalIndex is lost after reading back #19959
Wes McKinney / @wesm:
As has been discussed in other issues, this is a somewhat complex topic:
Parquet does not have a categorical type
Dictionary encoding is used in general for compression only
A single RowGroup may stop using dictionary encoding mid-stream if the dictionary grows large
The dictionary will be different from row group to roup group in general
In the case where both the writer and the reader is pandas, we can add pandas-specific metadata to indicate that the same dictionary has been written in each row group, and that all the data pages are dictionary encoded, to avoid the issues above. It's a doable project but with a steep learning curve, requiring comfort level with multiple layers of the C++ library and the Arrow conversion layer.
Matthew Rocklin / @mrocklin:
Right, I've just done a search on this topic and found a few other issues. Happy to move conversation to one of them if desired.
In this particular case we already have pandas-specific metadata telling us that we have a categorical column (this is in the pandas_type entry for the column). As you say we don't know the category-mapping, but we could still create a Categorical column and just let Pandas decide the mapping. I suspect that it sorts the entries as a sane default.. This wouldn't necessarily roundtrip the right category-to-code mapping, but we can at least roundtrip the fact that these should be represented as categoricals.
Speaking from a Dask Dataframe perspective this would be desirable. We're accustomed to having to remap mismatched categorical columns to make things match up. Pandas generally handles this case well today. However, I can understand if Arrow wants to play things safe here and resist the temptation to let Pandas be magical.
So, the concrete ask here is that when converting a table to a Pandas dataframe and using the use_pandas_metadata= keyword that we convert columns that Pandas marked with pandas_type='category' to be categorical columns.
Wes McKinney / @wesm:
Makes sense. We've recently done a bunch of work to improve the hashing machinery on the C++ side (for dictionary encoding, or whatever purpose). This has been a common enough issue / ask that I think it would be a good idea to tackle this in the next couple of months, so for the Arrow 0.13 release (0.12 is right around the corner)
Wes McKinney / @wesm:
I'm pretty close to being able to take care of this one finally. There are some Parquet-related oddities such as dealing with "large" categories if we want to accurately preserve categories in all cases.
Activity
asfimport commentedon Dec 31, 2018
Matthew Rocklin / @mrocklin:
This is lost for categorical columns generally, not only the index
asfimport commentedon Dec 31, 2018
Wes McKinney / @wesm:
As has been discussed in other issues, this is a somewhat complex topic:
Parquet does not have a categorical type
Dictionary encoding is used in general for compression only
A single RowGroup may stop using dictionary encoding mid-stream if the dictionary grows large
The dictionary will be different from row group to roup group in general
In the case where both the writer and the reader is pandas, we can add pandas-specific metadata to indicate that the same dictionary has been written in each row group, and that all the data pages are dictionary encoded, to avoid the issues above. It's a doable project but with a steep learning curve, requiring comfort level with multiple layers of the C++ library and the Arrow conversion layer.
asfimport commentedon Jan 1, 2019
Matthew Rocklin / @mrocklin:
Right, I've just done a search on this topic and found a few other issues. Happy to move conversation to one of them if desired.
In this particular case we already have pandas-specific metadata telling us that we have a categorical column (this is in the
pandas_typeentry for the column). As you say we don't know the category-mapping, but we could still create a Categorical column and just let Pandas decide the mapping. I suspect that it sorts the entries as a sane default.. This wouldn't necessarily roundtrip the right category-to-code mapping, but we can at least roundtrip the fact that these should be represented as categoricals.Speaking from a Dask Dataframe perspective this would be desirable. We're accustomed to having to remap mismatched categorical columns to make things match up. Pandas generally handles this case well today. However, I can understand if Arrow wants to play things safe here and resist the temptation to let Pandas be magical.
So, the concrete ask here is that when converting a table to a Pandas dataframe and using the
use_pandas_metadata=keyword that we convert columns that Pandas marked withpandas_type='category'to be categorical columns.asfimport commentedon Jan 1, 2019
Wes McKinney / @wesm:
Makes sense. We've recently done a bunch of work to improve the hashing machinery on the C++ side (for dictionary encoding, or whatever purpose). This has been a common enough issue / ask that I think it would be a good idea to tackle this in the next couple of months, so for the Arrow 0.13 release (0.12 is right around the corner)
asfimport commentedon Aug 2, 2019
Wes McKinney / @wesm:
I'm pretty close to being able to take care of this one finally. There are some Parquet-related oddities such as dealing with "large" categories if we want to accurately preserve categories in all cases.
asfimport commentedon Aug 19, 2019
Wes McKinney / @wesm:
This is working in master. I'm going to add a unit test
asfimport commentedon Aug 19, 2019
Wes McKinney / @wesm:
Issue resolved by pull request 5117
#5117