Skip to content

GH-33825: [Python] Expose pyarrow.dataset.get_partition_keys publicly (get key/value from partition expression) - #33862

Merged
jorisvandenbossche merged 3 commits into
apache:mainfrom
jorisvandenbossche:gh-33825-get_partition_keys
Apr 5, 2023
Merged

jorisvandenbossche merged 3 commits into
apache:mainfrom
jorisvandenbossche:gh-33825-get_partition_keys

Conversation

@jorisvandenbossche

@jorisvandenbossche jorisvandenbossche commented Jan 25, 2023 •

Copy link
Copy Markdown
Member

Rationale for this change

We have an existing "semi-private" pyarrow.dataset._get_partition_keys function (to get the partitioning field's key/value from the partition expression of a certain fragment). This is used by external projects (eg dask), and generally useful for advanced users, so let's just make it public.

…blicly (get key/value from partition expression)
@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #33825 has been automatically assigned in GitHub to PR creator.

@danepitkin

Copy link
Copy Markdown
Member

LGTM!

@amol-

amol- commented Mar 30, 2023

Copy link
Copy Markdown
Member

Closing because it has been untouched for a while, in case it's still relevant feel free to reopen and move it forward 👍

@jorisvandenbossche
jorisvandenbossche merged commit 2678db3 into apache:main Apr 5, 2023
@jorisvandenbossche
jorisvandenbossche deleted the gh-33825-get_partition_keys branch April 5, 2023 17:12
@ursabot

ursabot commented Apr 6, 2023

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 0484a21 and contender = 2678db3. 2678db3 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.51% ⬆️0.0%] ursa-i9-9960x
[Failed ⬇️0.0% ⬆️0.0%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 2678db3d ec2-t3-xlarge-us-east-2
[Failed] 2678db3d test-mac-arm
[Finished] 2678db3d ursa-i9-9960x
[Failed] 2678db3d ursa-thinkcentre-m75q
[Finished] 0484a21c ec2-t3-xlarge-us-east-2
[Failed] 0484a21c test-mac-arm
[Finished] 0484a21c ursa-i9-9960x
[Failed] 0484a21c ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ArgusLi pushed a commit to Bit-Quill/arrow that referenced this pull request May 15, 2023
…blicly (get key/value from partition expression) (apache#33862)

#### Rationale for this change

We have an existing "semi-private" `pyarrow.dataset._get_partition_keys` function (to get the partitioning field's key/value from the partition expression of a certain fragment). This is used by external projects (eg dask), and generally useful for advanced users, so let's just make it public. 

* Closes: apache#33825

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python] Introspect partition keys and values in fragments.

4 participants