You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++] Teach dataset_writer to accept custom functor on base file name template #34565
Hi,
I want to report a feature request as nowadays dataset_writer only supports basename_template with kIntegerToken. Say I have file 0.parquet, 1,parquet, ..., 10.parquet, 11.parquet and 12.parquet. It does not work with alphabetical sorter and downstream users must implement lexicographic sorter accordingly. In my case, I need to touch quite a few codebases to support hive style partition parquet in my org.
I propose to add a new option which allows left padding with zeros. so the names will be 001,parquet, ...,010.parquet, 011.parquet and 012.parquet. Let me know thoughts and suggestions as I can help with the contribution.
That seems like a reasonable ask. Would you suggest doing this with patterns in the basename template? I'm guessing the old approach {i} was based on python's f strings ({i: <16}) so maybe we should pick something like printf that is available in C++ and then fully support the various format strings.
Or, there are probably other ways to do this as well.
Hi Weston yes and thanks for the reply. I'm open to either C printf or pick something from STD. Is it ok to limit the scope to left padding in issue, or you have other suggestions?
If it is easier to limit the scope to left padding that is fine.
Using something from std is fine (and probably preferred) as well.
If we do use something from std/printf then we might find it is actually harder to limit the scope to left padding. So I also don't mind if we end up supporting something more.
I have some new thoughts on this issue. Instead of supporting just left padding, I propose we allow users to provide a lambda function with type as std::function(std::string<int>) in the WriterOption. By doing so, users can do whatever they want(e.g. left padding, use hash value as file name, etc) with a generic solution in C++. If that sounds good to you I will tackle it in my spare cycle in April.
changed the title [-][C++] dataset_writer does not support left padding[/-][+][C++] Teach dataset_writer to accept custom functor on base file name template[/+]on Apr 5, 2023
Activity
HaochengLIU commentedon Mar 16, 2023
Hi @westonpace , can you comment on this issue when you have time? Ty.
westonpace commentedon Mar 21, 2023
That seems like a reasonable ask. Would you suggest doing this with patterns in the basename template? I'm guessing the old approach
{i}was based on python's f strings ({i: <16}) so maybe we should pick something like printf that is available in C++ and then fully support the various format strings.Or, there are probably other ways to do this as well.
HaochengLIU commentedon Mar 21, 2023
Hi Weston yes and thanks for the reply. I'm open to either C printf or pick something from STD. Is it ok to limit the scope to left padding in issue, or you have other suggestions?
westonpace commentedon Mar 22, 2023
If it is easier to limit the scope to left padding that is fine.
Using something from std is fine (and probably preferred) as well.
If we do use something from std/printf then we might find it is actually harder to limit the scope to left padding. So I also don't mind if we end up supporting something more.
HaochengLIU commentedon Mar 22, 2023
Cool. I will ping you again once the PR is ready.
HaochengLIU commentedon Mar 31, 2023
I have some new thoughts on this issue. Instead of supporting just left padding, I propose we allow users to provide a lambda function with type as
std::function(std::string<int>)in the WriterOption. By doing so, users can do whatever they want(e.g. left padding, use hash value as file name, etc) with a generic solution in C++. If that sounds good to you I will tackle it in my spare cycle in April.westonpace commentedon Mar 31, 2023
That's a good idea!
[-][C++] dataset_writer does not support left padding[/-][+][C++] Teach dataset_writer to accept custom functor on base file name template[/+]HaochengLIU commentedon Apr 7, 2023
Hi weston can you guide how to run test dataset_writer_test? it keeps telling me 0 test gets run...
westonpace commentedon Apr 7, 2023
Sorry, that's due to a bug that was briefly checked into the main branch. Can you rebase the main branch? It should be fixed now.
HaochengLIU commentedon Apr 7, 2023
Rebasing fixes it now. Thanks Weston
GH-34565: [C++] Teach dataset_writer to accept custom filename functor (
apacheGH-34565: [C++] Teach dataset_writer to accept custom filename …
apacheGH-34565: [C++] Teach dataset_writer to accept custom filename …