Repository navigation
ARROW-3789: [Python] Use common conversion path for Arrow to pandas.Series/DataFrame. Zero copy optimizations for DataFrame, add split_blocks and self_destruct options - #6067
Conversation
|
I will rebase this and add docs. It's a little bit frustrating to have rebase conflicts due to a patch that only fixes typos =/ |
bb39b06 to
a85286e
Compare
|
I added docs, so aside from CI issues this is ready to go. @jorisvandenbossche can you help review this at least for the high level parts? |
|
I'll try to take a look at it too during the week, perhaps I'll also add a couple of hypothesis tests for pandas roundtrips. |
There was a problem hiding this comment.
Note that this doesn't exist (yet)
There was a problem hiding this comment.
| In the worst case scenario, calling ``to_pandas`` will result in a two versions | |
| In the worst case scenario, calling ``to_pandas`` will result in two versions |
There was a problem hiding this comment.
| In these scenarios, ``to_pandas`` will be zero copy. In all other scenarios, a | |
| In these scenarios, ``to_pandas`` or ``to_numpy`` will be zero copy. In all other scenarios, a |
?
There was a problem hiding this comment.
| As if this writing, pandas applies a data management strategy called | |
| As of this writing, pandas applies a data management strategy called |
There was a problem hiding this comment.
Maybe include a del table on a second line to show as good practice to make it clear that the table should not be used any more?
There was a problem hiding this comment.
| EXPERIMENTAL: If True, attempt to deallocate memory while | |
| EXPERIMENTAL: If True, attempt to deallocate the original Arrow memory while |
?
There was a problem hiding this comment.
That seems a bug in the current to_numpy that it returns a dictionary for dictionary type data.
Maybe can move this if block to to_numpy
There was a problem hiding this comment.
Will try to check this later (reminder for myself)
There was a problem hiding this comment.
Was this already tested?
There was a problem hiding this comment.
Seems it is tested here https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_extension_type.py#L394
zero-copy operations for improved memory use when converting from Arrow to pandas
|
Rebased and addressed comments. I'll merge this once the build looks okay |
|
Appveyor build: https://ci.appveyor.com/project/wesm/arrow/builds/30117011. Merging |
|
Can those parameters also be exposed to for example: |
|
Currently, they are only exposed in (in pandas.read_feather, that's something else, there it would be nice to have a way to pass through those keywords) But feel free to open a JIRA issue about it! (I am only one opinion) |
The primary goal of this patch is to provide a way for some users to avoid memory doubling with converting from Arrow to pandas.
This took me entirely too much time to get right, but partly I was attempting to disentangle some of the technical debt and overdue refactoring in arrow_to_pandas.cc.
Summary of what's here:
ArrowDeserializerhelper class is now gonesplit_blocksoption toto_pandaswhich constructs one block per DataFrame column, resulting in more zero-copy opportunities. Note that pandas's internal "consolidation" can still cause memory doubling (see discussion about this in Expose the blocks API and disable automatic consolidation pandas-dev/pandas#10556)self_destructoption toto_pandaswhich releases the Table's internal buffers as soon as they are converted to the required pandas structure. This allows memory to be reclaimed by the OS as conversion is taking place rather than having a forced memory-doubling and then post-facto reclamation (which has been causing OOM for some users)The most conservative invocation of
to_pandasnow would betable.to_pandas(use_threads=False, split_blocks=True, self_destruct=True)Note that the self-destruct option makes the
Tableobject unsafe for further use. This is a bit dissatisfying but I wasn't sure how else to provide this capability.