Repository navigation
GH-34565: [C++] Teach dataset_writer to accept custom filename functor - #34984
westonpace merged 1 commit into
Conversation
|
@westonpace Mind taking a look when you have time? Ty |
|
|
281bd70 to
c3f0fed
Compare
westonpace
left a comment
There was a problem hiding this comment.
A few small suggestions but this looks good. Thank you for the contribution!
…functor ### Rationale for this change Existing basename_template will only use a monotonically increasing int as new filenames. when there is needs for custom filenames(left padding, hash-code), downstream users must rename the files in a post processing step. ### What changes are included in this PR? A new functor is added to FileSystemDatasetWriteOptions which allows users to provide a custom name for dataset_writer. ### Are these changes tested? Yes. Unit tests are added for normal and ill formed lambdas. ### Are there any user-facing changes? Yes. It allows user to customize output file names. * Closes: apache#34565
c3f0fed to
3fc1938
Compare
Happy to pick up my long-last love of open-source and Arrow is a fantastic project. Ty for taking the time reviewing :) |
|
One quick question: Do you happen to know when Arrow 12.0.0 will be released? Would like to get this PR in so I could use at work |
We have a feature freeze policy for our Releases, see: https://arrow.apache.org/docs/dev/developers/release.html#creating-a-release-candidate |
|
Thanks for the info Raul. @westonpace If possible can I catch the last train :) ? The failing checks are not related to the PR |
|
It seems weston is on west coast. @raulcd Can you guide if we could wait until he is online? Ty |
Sure, I can wait slightly longer. I'll try to ping him if there's no answer but in order to follow the current schedule for the release I'll branch the 12.0.0 release today (CEST timezone). |
|
SGTM. Thanks Raul |
|
Benchmark runs are scheduled for baseline = 1912fb9 and contender = 88fd523. 88fd523 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
|
['Python', 'R'] benchmarks have high level of regressions. |
…functor (apache#34984) ### Rationale for this change Existing basename_template will only use a monotonically increasing int as new filenames. when there is needs for custom filenames(left padding, hash-code), downstream users must rename the files in a post-processing step. ### What changes are included in this PR? A new functor is added to FileSystemDatasetWriteOptions which allows users to provide a custom name for dataset_writer. ### Are these changes tested? Yes. Unit tests are added for normal and ill-formed lambdas. ### Are there any user-facing changes? Yes. It allows users to customize output file names. * Closes: apache#34565 Authored-by: Haocheng Liu <lbtinglb@gmail.com> Signed-off-by: Weston Pace <weston.pace@gmail.com>
…functor (apache#34984) ### Rationale for this change Existing basename_template will only use a monotonically increasing int as new filenames. when there is needs for custom filenames(left padding, hash-code), downstream users must rename the files in a post-processing step. ### What changes are included in this PR? A new functor is added to FileSystemDatasetWriteOptions which allows users to provide a custom name for dataset_writer. ### Are these changes tested? Yes. Unit tests are added for normal and ill-formed lambdas. ### Are there any user-facing changes? Yes. It allows users to customize output file names. * Closes: apache#34565 Authored-by: Haocheng Liu <lbtinglb@gmail.com> Signed-off-by: Weston Pace <weston.pace@gmail.com>
Rationale for this change
Existing basename_template will only use a monotonically increasing int as new filenames. when there is needs for custom filenames(left padding, hash-code), downstream users must rename the files in a post-processing step.
What changes are included in this PR?
A new functor is added to FileSystemDatasetWriteOptions which allows users to provide a custom name for dataset_writer.
Are these changes tested?
Yes. Unit tests are added for normal and ill-formed lambdas.
Are there any user-facing changes?
Yes. It allows users to customize output file names.