Skip to content

ARROW-1705: [Python] allow building array from dicts - #1530

Closed
pitrou wants to merge 1 commit into
apache:masterfrom
pitrou:ARROW-1705-struct-array-from-dicts
Closed

pitrou wants to merge 1 commit into
apache:masterfrom
pitrou:ARROW-1705-struct-array-from-dicts

Conversation

@pitrou

@pitrou pitrou commented Jan 29, 2018

Copy link
Copy Markdown
Member

Accept passing a list of dicts to pa.array() if a struct type is given.

Based on PR #1513.

@pitrou
pitrou force-pushed the ARROW-1705-struct-array-from-dicts branch from 9828983 to e984215 Compare January 29, 2018 17:43

@wesm wesm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will review in more detail after rebasing once ARROW-2033 goes in

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this virtual has performance implications because of inlining (this module is using CRTP to simulate virtual functions without the dynamic dispatch); in practice it may not matter, though

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this PR is not ondoing the CRTP trick, so I don't think it has performance implications?

I was adding this for two reasons:

  1. for consistency with AppendNull which has a default version defined on TypedConverterVisitor (it only needs to be specialized for struct types)
  2. for documentation, since it makes it more obvious that subclasses are supposed to define AppendItem

Should I remove it? Or is there a way to have a non-virtual unimplemented base method?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is OK as is; when we can demonstrate real performance benefits to using non-virtual calls we can do some refactoring later

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, actually you may be right that even the static_cast<> won't be enough to devirtualize the method call (since we may be given a derived type defined somewhere).

I ran a simple micro-benchmark:

$ python -m timeit -s "import pyarrow as pa; ty=pa.int64(); data=[42]*1000000" "pa.array(data, type=ty)"
  • before: 10 loops, best of 3: 42.4 msec per loop
  • after: 10 loops, best of 3: 45.5 msec per loop

So there is a slight slowdown indeed...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I undid this. We're now down to 43.3 msec per loop in the micro-benchmark above (so a tiny bit slower than master, but presumably good enough).

@pitrou
pitrou force-pushed the ARROW-1705-struct-array-from-dicts branch from 4592431 to 847ec80 Compare January 30, 2018 12:15
@pitrou pitrou changed the title [WIP] ARROW-1705: [Python] allow building array from dicts ARROW-1705: [Python] allow building array from dicts Jan 30, 2018
@wesm
wesm force-pushed the ARROW-1705-struct-array-from-dicts branch from 847ec80 to ba650fe Compare January 30, 2018 15:20
@wesm

wesm commented Jan 30, 2018

Copy link
Copy Markdown
Member

Rebased

@pitrou

pitrou commented Jan 30, 2018

Copy link
Copy Markdown
Member Author

Did you use a specific git incantation for rebasing? Or do you just fix all conflicts by hand?

@wesm

wesm commented Jan 30, 2018 •

Copy link
Copy Markdown
Member

In this case I had to fix by hand. What I usually do in the case of interdependent branches:

  • Base dependent branch B on a git rebase -i version of base branch A
  • Branch A gets merged
  • Branch B will now git rebase cleanly

This works well as a result of the way we are merging PRs (by always rolling them up into an atomic patch)

@wesm
wesm force-pushed the ARROW-1705-struct-array-from-dicts branch from ba650fe to 7120196 Compare January 30, 2018 15:32
@pitrou
pitrou force-pushed the ARROW-1705-struct-array-from-dicts branch 2 times, most recently from d9c0324 to 3ca7e5d Compare January 30, 2018 15:48
@wesm

wesm commented Jan 30, 2018

Copy link
Copy Markdown
Member

Sorry, I messed things up (I fixed the commit authorship in master)

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is OK as is; when we can demonstrate real performance benefits to using non-virtual calls we can do some refactoring later

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point we may set null if a particular key is not found, raising KeyError seems OK right now

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyDict_GetItem doesn't raise KeyError, it simply returns NULL when the key isn't found (see also the tests which exercise this case).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted, thank you =)

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how much of a measurable performance impact it will have (it would be good to know for future reference), but I made a separate vector of raw pointers to the internal builders for RecordBatchBuilder to avoid the extra shared_ptr overhead on the inner loop https://github.com/apache/arrow/blob/master/cpp/src/arrow/table_builder.h#L105

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll give a try.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I tried it with the following micro-benchmark:

$ python -m timeit -s "import pyarrow as pa; ty=pa.struct([pa.field('x', pa.int32())]); data=[None]*1000000" "pa.array(data, type=ty)"
  • unpatched: 49.3 msec per loop
  • with unique_ptr instead of shared_ptr: 44.3 msec per loop
  • with raw pointers in addition to shared_ptr: 43.6 msec per loop

Since an additional array of raw pointers adds some complication, I'm tempted to go with the unique_ptr solution (which AFAICS looks sane).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I migrated the code to unique_ptr.

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DCHECK_EQ

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a test that raises a KeyError for a missing field?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it doesn't, that's the point... unless I'm misunderstanding you here?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're all good (per my misunderstanding about PyDict_GetItem)

@pitrou
pitrou force-pushed the ARROW-1705-struct-array-from-dicts branch from 3ca7e5d to ca68ed2 Compare January 30, 2018 16:13
Accept passing a list of dicts to pa.array() if a struct type is given.
@pitrou
pitrou force-pushed the ARROW-1705-struct-array-from-dicts branch from ca68ed2 to 2b9133a Compare January 30, 2018 16:14
@wesm

wesm commented Jan 30, 2018

Copy link
Copy Markdown
Member

+1, this looks good to go once the build runs. thanks @pitrou!

@pitrou

pitrou commented Jan 30, 2018

Copy link
Copy Markdown
Member Author

(as a side note, who has the right to retry failed jobs on Travis? there are apt failures again :-))

@wesm

wesm commented Jan 30, 2018

Copy link
Copy Markdown
Member

Folks with write access to the git repo (me, Uwe, @cpcloud, others) -- anyone on http://arrow.apache.org/committers/. I restarted the jobs. Not sure why apt on Travis has been so flaky lately

@wesm

wesm commented Jan 30, 2018

Copy link
Copy Markdown
Member

Restarted the failed Travis CI build. Can you enable Appveyor on your fork so we can potentially get some faster builds to look at there? Thanks!

@pitrou

pitrou commented Jan 30, 2018

Copy link
Copy Markdown
Member Author

Ok, there's an AppVeyor build starting at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.2

@wesm wesm closed this in 5c704bc Jan 30, 2018
@pitrou
pitrou deleted the ARROW-1705-struct-array-from-dicts branch January 31, 2018 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants