Skip to content

[C++] The C++ API for writing datasets could be improved #30891

Closed as not planned

Description

@asfimport

I was working on write dataset testing in the C++ API today and ran into a number of things that were not very intuitive. All of these are abstracted away / hidden by the python / R interface so this really only applies to anyone using the C++ API directly.

  • If no partitioning is specified the write will segfault. Instead it should us a default (no-op) partitioning.
  • The min_rows_per_group option should probably default to something higher than 0
  • It's not clear how to specify the format (you do it by creating a format, then setting the file write options, which sets the format privately)
  • There is no default for basename_template
  • There is no default for filesystem (should be local filesystem)

Reporter: Weston Pace / @westonpace

Related issues:

PRs and other links:

Note: This issue was originally created as ARROW-15409. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Nov 29, 2022

    @asfimport
    Author

    Apache Arrow JIRA Bot:
    This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

  2. vibhatha commented on Jan 13, 2023

    @vibhatha
    Contributor

    @westonpace there is an issue in defining the format. Because all the format definitions depends on the definition on this file_base.h base class. It is not very clear how to set this value.

    Also note that looking into the previous PR: #13959

    If min_rows_per_group value is set to a higher default value and if we set the max_rows_per_group, and in an event of max_rows_per_group < min_rows_per_group, it fails an existing assertion which make sense. So even in a simpler case where user just needs to set the max value, they may be forced to set the min value.

    So I would say keeping it 0 as default would be okay. Am I missing any thing here?

    Appreciate your thoughts on these two points.

  3. westonpace commented on Jan 13, 2023

    @westonpace
    Member

    @westonpace there is an issue in defining the format. Because all the format definitions depends on the definition on this file_base.h base class. It is not very clear how to set this value.

    Make the format a required constructor property. Get rid of the no-arg constructor.

    So I would say keeping it 0 as default would be okay. Am I missing any thing here?

    A default of 1Mi for min_rows_per_group and 64Mi for max_rows_per_group should be very reasonable and give better performance than leaving it at 0 in many cases.

  4. kou commented on Mar 31, 2023

    @kou
    Member

    If no partitioning is specified the write will segfault. Instead it should us a default (no-op) partitioning.

    This will be resolved by #15256.

  5. vibhatha commented on Apr 1, 2023

    @vibhatha
    Contributor

    Should we close this. @kou

  6. kou commented on Apr 1, 2023

    @kou
    Member

    I don't think so because at least #15256 doesn't solve "There is no default for filesystem (should be local filesystem)".

  7. removed their assignment
    on Jun 2, 2024
  8. github-actions commented on Mar 8, 2026

    @github-actions

    This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

  9. added
    Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
    on Mar 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions