Skip to content

[C++] Create DataHolder that can be used for caching during exec plans #29901

Closed as not planned

Description

@asfimport

The purpose of this task is to make an ExecNode that can provide the following functionality.

  1. Be able to obtain heuristics about our memory consumption and have a memory consumption threshold

  2. Be able to write incoming ExecBatch to disk if memory consumption is above the threshold, stores either the ExecBatch or a handle to file in a queue.

  3. Provide an api for pulling an ExecBatch from the queue. It should favor pulling all of the batches that are in memory first and then the ones that are handles to files.

     

    PRs to reference

    ARROW-13542: [C++][Compute][Dataset] Add dataset::WriteNode for writing rows from an ExecPlan to disk #11017

     

Reporter: Alexander Ocsa / @aocsa

PRs and other links:

Note: This issue was originally created as ARROW-14330. Please see the migration documentation for further details.

Activity

  1. added
    Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
    and removed on Sep 23, 2026
  2. thisisnic commented on Sep 23, 2026

    @thisisnic
    Member

    This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

  3. removed
    Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
    on Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions