You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++] The C++ API for writing datasets could be improved #30891
I was working on write dataset testing in the C++ API today and ran into a number of things that were not very intuitive. All of these are abstracted away / hidden by the python / R interface so this really only applies to anyone using the C++ API directly.
If no partitioning is specified the write will segfault. Instead it should us a default (no-op) partitioning.
The min_rows_per_group option should probably default to something higher than 0
It's not clear how to specify the format (you do it by creating a format, then setting the file write options, which sets the format privately)
There is no default for basename_template
There is no default for filesystem (should be local filesystem)
Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.
@westonpace there is an issue in defining the format. Because all the format definitions depends on the definition on this file_base.h base class. It is not very clear how to set this value.
Also note that looking into the previous PR: #13959
If min_rows_per_group value is set to a higher default value and if we set the max_rows_per_group, and in an event of max_rows_per_group < min_rows_per_group, it fails an existing assertion which make sense. So even in a simpler case where user just needs to set the max value, they may be forced to set the min value.
So I would say keeping it 0 as default would be okay. Am I missing any thing here?
@westonpace there is an issue in defining the format. Because all the format definitions depends on the definition on this file_base.h base class. It is not very clear how to set this value.
Make the format a required constructor property. Get rid of the no-arg constructor.
So I would say keeping it 0 as default would be okay. Am I missing any thing here?
A default of 1Mi for min_rows_per_group and 64Mi for max_rows_per_group should be very reasonable and give better performance than leaving it at 0 in many cases.
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
Activity
asfimport commentedon Nov 29, 2022
Apache Arrow JIRA Bot:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.
vibhatha commentedon Jan 13, 2023
@westonpace there is an issue in defining the
format. Because all the format definitions depends on the definition on thisfile_base.hbase class. It is not very clear how to set this value.Also note that looking into the previous PR: #13959
If
min_rows_per_groupvalue is set to a higher default value and if we set themax_rows_per_group, and in an event ofmax_rows_per_group<min_rows_per_group, it fails an existing assertion which make sense. So even in a simpler case where user just needs to set the max value, they may be forced to set the min value.So I would say keeping it 0 as default would be okay. Am I missing any thing here?
Appreciate your thoughts on these two points.
westonpace commentedon Jan 13, 2023
Make the format a required constructor property. Get rid of the no-arg constructor.
A default of 1Mi for
min_rows_per_groupand 64Mi formax_rows_per_groupshould be very reasonable and give better performance than leaving it at0in many cases.kou commentedon Mar 31, 2023
This will be resolved by #15256.
vibhatha commentedon Apr 1, 2023
Should we close this. @kou
kou commentedon Apr 1, 2023
I don't think so because at least #15256 doesn't solve "There is no default for filesystem (should be local filesystem)".
github-actions commentedon Mar 8, 2026
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.