You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Python] Introspect partition keys and values in fragments. #33825
It's not possible to programmatically determine the values of partition keys in a fragment. Fragments have a partition_expression attribute, but the Expression type doesn't allow any further introspection. I don't want to have to parse the string representation of the expression.
In []: dataset.partitioning.schemaOut[]:
year: int32month: int32In []: fragment=next(dataset.get_fragments())
In []: fragment.partition_expressionOut[]: <pyarrow.compute.Expression ((year==2013) and (month==1))>
My broader use case is more performant (speed and memory) aggregation of partitioned data. Using pc._group_by requires loaded arrays, so it ignores that the data is already partitioned. And iterating get_fragments is crippled if one can't identify the fragment.
There is _get_partition_keys which is private, but has been kept around. I think there were past discussions about promoting it to a public API.
Yes, we should probably do that (we did the same for pyarrow.parquet.filters_to_expression). I think we can re-scope this issue to do that.
Also @coady feel free to already use the current "private" method. It's private because it was thought to not be really user-facing, we know it is used (eg dask uses it as well), so we promise some stability for it.
Something related that might be worth to mention (not that it solves your exact use case here though): there is also a dataset.partitioning.dictionaries if you want to inspect all possible values for all fragments that a certain partitioning field can be.
Activity
lidavidm commentedon Jan 21, 2023
There is
_get_partition_keyswhich is private, but has been kept around. I think there were past discussions about promoting it to a public API.westonpace commentedon Jan 22, 2023
That's an interesting broader use case. Can you add a new GH issue for the broader ask of a simpler way to do aggregations on partitions?
jorisvandenbossche commentedon Jan 25, 2023
Yes, we should probably do that (we did the same for
pyarrow.parquet.filters_to_expression). I think we can re-scope this issue to do that.Also @coady feel free to already use the current "private" method. It's private because it was thought to not be really user-facing, we know it is used (eg dask uses it as well), so we promise some stability for it.
jorisvandenbossche commentedon Jan 25, 2023
Something related that might be worth to mention (not that it solves your exact use case here though): there is also a
dataset.partitioning.dictionariesif you want to inspect all possible values for all fragments that a certain partitioning field can be.apacheGH-33825: [Python] Expose pyarrow.dataset.get_partition_keys pu…
jorisvandenbossche commentedon Jan 25, 2023
Opened #33862 to make it public.
Merge remote-tracking branch 'upstream/main' into apachegh-33825-get_…
GH-33825: [Python] Expose pyarrow.dataset.get_partition_keys publicly…
apacheGH-33825: [Python] Expose pyarrow.dataset.get_partition_keys pu…