Skip to content

[Python] Enable calling object in Table.to_pandas to "self-destruct" for improved memory use #15871

Closed

Description

@asfimport

One issue with using Table.to_pandas is that it results in a memory doubling (at least, more if there are a lot of Python objects created). It would be useful if there was an option to destroy the arrow::Column references once they've been transferred into the target data frame. This would render the pyarrow.Table object useless afterward

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

PRs and other links:

Note: This issue was originally created as ARROW-3789. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Nov 6, 2019

    @asfimport
    Author

    Wes McKinney / @wesm:
    There's a few things that have to happen:

    • Add "self-destruct" option that can result in the passed table's columns being deallocated as they are consumed. This should warn the user if for some reason the shared_ptr use_count is not 1, so that users will know that the self-destructing behavior is not taking place

    • Add "split blocks" option to create one block per column, to limit memory doubling to a per-column basis rather than per-data type. In the worst case, we might have a table with all columns having the same data type – the current block creation behavior would yield memory doubling even with the column "self-destructing" described above

    • Some split blocks can be zero-copy constructed when there are no nulls

    • Users will need to be advised to set pandas to use a no-consolidation split block policy

      I'm looking at arrow_to_pandas.cc to come up with a plan to limit code duplication between the split/non-split block construction path

  2. asfimport commented on Jan 15, 2020

    @asfimport
    Author

    Wes McKinney / @wesm:
    Issue resolved by pull request 6067
    #6067

  3. added this to the 0.16.0 milestone on Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions