Skip to content

GH-34565: [C++] Teach dataset_writer to accept custom filename functor - #34984

Merged
westonpace merged 1 commit into
apache:mainfrom
HaochengLIU:teach-dataset_writer-to-accept-custom-functor
Apr 12, 2023
Merged

westonpace merged 1 commit into
apache:mainfrom
HaochengLIU:teach-dataset_writer-to-accept-custom-functor

Conversation

@HaochengLIU

@HaochengLIU HaochengLIU commented Apr 8, 2023 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Existing basename_template will only use a monotonically increasing int as new filenames. when there is needs for custom filenames(left padding, hash-code), downstream users must rename the files in a post-processing step.

What changes are included in this PR?

A new functor is added to FileSystemDatasetWriteOptions which allows users to provide a custom name for dataset_writer.

Are these changes tested?

Yes. Unit tests are added for normal and ill-formed lambdas.

Are there any user-facing changes?

Yes. It allows users to customize output file names.

@HaochengLIU
HaochengLIU requested a review from westonpace as a code owner April 8, 2023 04:55
@github-actions

github-actions Bot commented Apr 8, 2023

Copy link
Copy Markdown

@HaochengLIU

Copy link
Copy Markdown
Contributor Author

@westonpace Mind taking a look when you have time? Ty

@github-actions

github-actions Bot commented Apr 8, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #34565 has been automatically assigned in GitHub to PR creator.

@HaochengLIU
HaochengLIU force-pushed the teach-dataset_writer-to-accept-custom-functor branch from 281bd70 to c3f0fed Compare April 8, 2023 04:58

@westonpace westonpace left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small suggestions but this looks good. Thank you for the contribution!

Comment thread cpp/src/arrow/dataset/dataset_writer.cc Resolved
Comment thread cpp/src/arrow/dataset/dataset_writer.cc Resolved
Comment thread cpp/src/arrow/dataset/file_base.h Resolved
…functor

### Rationale for this change

Existing basename_template will only use a monotonically increasing int
as new filenames. when there is needs for custom filenames(left padding,
hash-code), downstream users must rename the files in a post processing
step.

### What changes are included in this PR?

A new functor is added to FileSystemDatasetWriteOptions which allows
users to provide a custom name for dataset_writer.

### Are these changes tested?

Yes. Unit tests are added for normal and ill formed lambdas.

### Are there any user-facing changes?

Yes. It allows user to customize output file names.
* Closes: apache#34565
@HaochengLIU
HaochengLIU force-pushed the teach-dataset_writer-to-accept-custom-functor branch from c3f0fed to 3fc1938 Compare April 12, 2023 01:25
@HaochengLIU

HaochengLIU commented Apr 12, 2023 •

Copy link
Copy Markdown
Contributor Author

A few small suggestions but this looks good. Thank you for the contribution!
PR updated.

Happy to pick up my long-last love of open-source and Arrow is a fantastic project. Ty for taking the time reviewing :)

@HaochengLIU

HaochengLIU commented Apr 12, 2023 •

Copy link
Copy Markdown
Contributor Author

One quick question: Do you happen to know when Arrow 12.0.0 will be released? Would like to get this PR in so I could use at work

@raulcd

raulcd commented Apr 12, 2023

Copy link
Copy Markdown
Member

One quick question: Do you happen to know when Arrow 12.0.0 will be released? Would like to get this PR in so I could use at work

We have a feature freeze policy for our Releases, see: https://arrow.apache.org/docs/dev/developers/release.html#creating-a-release-candidate
The code freeze is scheduled to be performed today. I plan to do it in the next couple of hours.

@HaochengLIU

HaochengLIU commented Apr 12, 2023 •

Copy link
Copy Markdown
Contributor Author

Thanks for the info Raul. @westonpace If possible can I catch the last train :) ?

The failing checks are not related to the PR

@HaochengLIU

Copy link
Copy Markdown
Contributor Author

It seems weston is on west coast. @raulcd Can you guide if we could wait until he is online? Ty

@raulcd

raulcd commented Apr 12, 2023

Copy link
Copy Markdown
Member

It seems weston is on west coast. @raulcd Can you guide if we could wait until he is online? Ty

Sure, I can wait slightly longer. I'll try to ping him if there's no answer but in order to follow the current schedule for the release I'll branch the 12.0.0 release today (CEST timezone).

@HaochengLIU

Copy link
Copy Markdown
Contributor Author

SGTM. Thanks Raul

@westonpace
westonpace merged commit 88fd523 into apache:main Apr 12, 2023
@ursabot

ursabot commented Apr 14, 2023

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 1912fb9 and contender = 88fd523. 88fd523 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Finished ⬇️3.57% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.34% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 88fd5237 ec2-t3-xlarge-us-east-2
[Failed] 88fd5237 test-mac-arm
[Finished] 88fd5237 ursa-i9-9960x
[Finished] 88fd5237 ursa-thinkcentre-m75q
[Finished] 1912fb9a ec2-t3-xlarge-us-east-2
[Failed] 1912fb9a test-mac-arm
[Finished] 1912fb9a ursa-i9-9960x
[Finished] 1912fb9a ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@ursabot

ursabot commented Apr 14, 2023

Copy link
Copy Markdown

['Python', 'R'] benchmarks have high level of regressions.
ursa-i9-9960x

liujiacheng777 pushed a commit to LoongArch-Python/arrow that referenced this pull request May 11, 2023
…functor (apache#34984)

### Rationale for this change

Existing basename_template will only use a monotonically increasing int as new filenames. when there is needs for custom filenames(left padding, hash-code), downstream users must rename the files in a post-processing step.

### What changes are included in this PR?

A new functor is added to FileSystemDatasetWriteOptions which allows users to provide a custom name for dataset_writer.

### Are these changes tested?

Yes. Unit tests are added for normal and ill-formed lambdas.

### Are there any user-facing changes?

Yes. It allows users to customize output file names.
* Closes: apache#34565

Authored-by: Haocheng Liu <lbtinglb@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
ArgusLi pushed a commit to Bit-Quill/arrow that referenced this pull request May 15, 2023
…functor (apache#34984)

### Rationale for this change

Existing basename_template will only use a monotonically increasing int as new filenames. when there is needs for custom filenames(left padding, hash-code), downstream users must rename the files in a post-processing step.

### What changes are included in this PR?

A new functor is added to FileSystemDatasetWriteOptions which allows users to provide a custom name for dataset_writer.

### Are these changes tested?

Yes. Unit tests are added for normal and ill-formed lambdas.

### Are there any user-facing changes?

Yes. It allows users to customize output file names.
* Closes: apache#34565

Authored-by: Haocheng Liu <lbtinglb@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Teach dataset_writer to accept custom functor on base file name template

4 participants