You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Python] Enable calling object in Table.to_pandas to "self-destruct" for improved memory use #15871
One issue with using Table.to_pandas is that it results in a memory doubling (at least, more if there are a lot of Python objects created). It would be useful if there was an option to destroy the arrow::Column references once they've been transferred into the target data frame. This would render the pyarrow.Table object useless afterward
Add "self-destruct" option that can result in the passed table's columns being deallocated as they are consumed. This should warn the user if for some reason the shared_ptr use_count is not 1, so that users will know that the self-destructing behavior is not taking place
Add "split blocks" option to create one block per column, to limit memory doubling to a per-column basis rather than per-data type. In the worst case, we might have a table with all columns having the same data type – the current block creation behavior would yield memory doubling even with the column "self-destructing" described above
Some split blocks can be zero-copy constructed when there are no nulls
Users will need to be advised to set pandas to use a no-consolidation split block policy
I'm looking at arrow_to_pandas.cc to come up with a plan to limit code duplication between the split/non-split block construction path
Activity
asfimport commentedon Nov 6, 2019
Wes McKinney / @wesm:
There's a few things that have to happen:
Add "self-destruct" option that can result in the passed table's columns being deallocated as they are consumed. This should warn the user if for some reason the shared_ptr use_count is not 1, so that users will know that the self-destructing behavior is not taking place
Add "split blocks" option to create one block per column, to limit memory doubling to a per-column basis rather than per-data type. In the worst case, we might have a table with all columns having the same data type – the current block creation behavior would yield memory doubling even with the column "self-destructing" described above
Some split blocks can be zero-copy constructed when there are no nulls
Users will need to be advised to set pandas to use a no-consolidation split block policy
I'm looking at arrow_to_pandas.cc to come up with a plan to limit code duplication between the split/non-split block construction path
asfimport commentedon Jan 15, 2020
Wes McKinney / @wesm:
Issue resolved by pull request 6067
#6067