Skip to content

ARROW-3789: [Python] Use common conversion path for Arrow to pandas.Series/DataFrame. Zero copy optimizations for DataFrame, add split_blocks and self_destruct options - #6067

Closed
wesm wants to merge 4 commits into
apache:masterfrom
wesm:ARROW-3789

Conversation

@wesm

@wesm wesm commented Dec 19, 2019 •

Copy link
Copy Markdown
Member

The primary goal of this patch is to provide a way for some users to avoid memory doubling with converting from Arrow to pandas.

This took me entirely too much time to get right, but partly I was attempting to disentangle some of the technical debt and overdue refactoring in arrow_to_pandas.cc.

Summary of what's here:

  • Refactor ChunkedArray->Series and Table->DataFrame conversion paths to use the exact same code rather than two implementations of the same thing with slightly different behavior. The ArrowDeserializer helper class is now gone
  • Do zero-copy construction of internal DataFrame blocks for the case of a contiguous non-nullable array and a block with only 1 column represented
  • Add split_blocks option to to_pandas which constructs one block per DataFrame column, resulting in more zero-copy opportunities. Note that pandas's internal "consolidation" can still cause memory doubling (see discussion about this in Expose the blocks API and disable automatic consolidation pandas-dev/pandas#10556)
  • Add self_destruct option to to_pandas which releases the Table's internal buffers as soon as they are converted to the required pandas structure. This allows memory to be reclaimed by the OS as conversion is taking place rather than having a forced memory-doubling and then post-facto reclamation (which has been causing OOM for some users)

The most conservative invocation of to_pandas now would be table.to_pandas(use_threads=False, split_blocks=True, self_destruct=True)

Note that the self-destruct option makes the Table object unsafe for further use. This is a bit dissatisfying but I wasn't sure how else to provide this capability.

@github-actions

Copy link
Copy Markdown

@wesm

wesm commented Jan 6, 2020

Copy link
Copy Markdown
Member Author

I will rebase this and add docs. It's a little bit frustrating to have rebase conflicts due to a patch that only fixes typos =/

@wesm
wesm force-pushed the ARROW-3789 branch 2 times, most recently from bb39b06 to a85286e Compare January 8, 2020 22:43
@wesm

wesm commented Jan 8, 2020

Copy link
Copy Markdown
Member Author

I added docs, so aside from CI issues this is ready to go. @jorisvandenbossche can you help review this at least for the high level parts?

@kszucs

kszucs commented Jan 8, 2020

Copy link
Copy Markdown
Member

I'll try to take a look at it too during the week, perhaps I'll also add a couple of hypothesis tests for pandas roundtrips.

@jorisvandenbossche jorisvandenbossche left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't review the C++ code in arrow_to_pandas.cc in detail, but read through it, and could generally follow the flow and logic.
And added some comments on the docs / python part.

Comment thread cpp/src/arrow/python/arrow_to_pandas.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this doesn't exist (yet)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread docs/source/python/pandas.rst Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In the worst case scenario, calling ``to_pandas`` will result in a two versions
In the worst case scenario, calling ``to_pandas`` will result in two versions

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread docs/source/python/pandas.rst Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
In these scenarios, ``to_pandas`` will be zero copy. In all other scenarios, a
In these scenarios, ``to_pandas`` or ``to_numpy`` will be zero copy. In all other scenarios, a

?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread docs/source/python/pandas.rst Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
As if this writing, pandas applies a data management strategy called
As of this writing, pandas applies a data management strategy called

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment thread docs/source/python/pandas.rst Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe include a del table on a second line to show as good practice to make it clear that the table should not be used any more?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea, done

Comment thread python/pyarrow/array.pxi Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
EXPERIMENTAL: If True, attempt to deallocate memory while
EXPERIMENTAL: If True, attempt to deallocate the original Arrow memory while

?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread python/pyarrow/array.pxi Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems a bug in the current to_numpy that it returns a dictionary for dictionary type data.

Maybe can move this if block to to_numpy

Comment thread python/pyarrow/array.pxi Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will try to check this later (reminder for myself)

Comment thread python/pyarrow/table.pxi Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this already tested?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wesm

wesm commented Jan 14, 2020

Copy link
Copy Markdown
Member Author

Rebased and addressed comments. I'll merge this once the build looks okay

@wesm

wesm commented Jan 15, 2020

Copy link
Copy Markdown
Member Author

Appveyor build: https://ci.appveyor.com/project/wesm/arrow/builds/30117011. Merging

@wesm wesm closed this in 3b000b7 Jan 15, 2020
@wesm
wesm deleted the ARROW-3789 branch January 15, 2020 00:25
@ghuls

ghuls commented Mar 9, 2020

Copy link
Copy Markdown
Contributor

Can those parameters also be exposed to for example: FeatherReader.read_pandas, read_feather and other functions where it makes sense?

@jorisvandenbossche

Copy link
Copy Markdown
Member

Currently, they are only exposed in to_table itself, and not in methods or functions that use to_table under the hood.
Personally, I would prefer people using feather.red_table(..).to_pandas(..), in which it is explicit which keywords belong to which conversion.

(in pandas.read_feather, that's something else, there it would be nice to have a way to pass through those keywords)

But feel free to open a JIRA issue about it! (I am only one opinion)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants