Repository navigation
Conversation
9828983 to
e984215
Compare
wesm
left a comment
There was a problem hiding this comment.
I will review in more detail after rebasing once ARROW-2033 goes in
There was a problem hiding this comment.
Making this virtual has performance implications because of inlining (this module is using CRTP to simulate virtual functions without the dynamic dispatch); in practice it may not matter, though
There was a problem hiding this comment.
Actually this PR is not ondoing the CRTP trick, so I don't think it has performance implications?
I was adding this for two reasons:
- for consistency with
AppendNullwhich has a default version defined onTypedConverterVisitor(it only needs to be specialized for struct types) - for documentation, since it makes it more obvious that subclasses are supposed to define
AppendItem
Should I remove it? Or is there a way to have a non-virtual unimplemented base method?
There was a problem hiding this comment.
This is OK as is; when we can demonstrate real performance benefits to using non-virtual calls we can do some refactoring later
There was a problem hiding this comment.
Hmm, actually you may be right that even the static_cast<> won't be enough to devirtualize the method call (since we may be given a derived type defined somewhere).
I ran a simple micro-benchmark:
$ python -m timeit -s "import pyarrow as pa; ty=pa.int64(); data=[42]*1000000" "pa.array(data, type=ty)"
- before: 10 loops, best of 3: 42.4 msec per loop
- after: 10 loops, best of 3: 45.5 msec per loop
So there is a slight slowdown indeed...
There was a problem hiding this comment.
Ok, I undid this. We're now down to 43.3 msec per loop in the micro-benchmark above (so a tiny bit slower than master, but presumably good enough).
4592431 to
847ec80
Compare
847ec80 to
ba650fe
Compare
|
Rebased |
|
Did you use a specific git incantation for rebasing? Or do you just fix all conflicts by hand? |
|
In this case I had to fix by hand. What I usually do in the case of interdependent branches:
This works well as a result of the way we are merging PRs (by always rolling them up into an atomic patch) |
ba650fe to
7120196
Compare
d9c0324 to
3ca7e5d
Compare
|
Sorry, I messed things up (I fixed the commit authorship in master) |
There was a problem hiding this comment.
This is OK as is; when we can demonstrate real performance benefits to using non-virtual calls we can do some refactoring later
There was a problem hiding this comment.
At some point we may set null if a particular key is not found, raising KeyError seems OK right now
There was a problem hiding this comment.
PyDict_GetItem doesn't raise KeyError, it simply returns NULL when the key isn't found (see also the tests which exercise this case).
There was a problem hiding this comment.
I'm not sure how much of a measurable performance impact it will have (it would be good to know for future reference), but I made a separate vector of raw pointers to the internal builders for RecordBatchBuilder to avoid the extra shared_ptr overhead on the inner loop https://github.com/apache/arrow/blob/master/cpp/src/arrow/table_builder.h#L105
There was a problem hiding this comment.
Ok, I tried it with the following micro-benchmark:
$ python -m timeit -s "import pyarrow as pa; ty=pa.struct([pa.field('x', pa.int32())]); data=[None]*1000000" "pa.array(data, type=ty)"
- unpatched: 49.3 msec per loop
- with unique_ptr instead of shared_ptr: 44.3 msec per loop
- with raw pointers in addition to shared_ptr: 43.6 msec per loop
Since an additional array of raw pointers adds some complication, I'm tempted to go with the unique_ptr solution (which AFAICS looks sane).
There was a problem hiding this comment.
Ok, I migrated the code to unique_ptr.
There was a problem hiding this comment.
Add a test that raises a KeyError for a missing field?
There was a problem hiding this comment.
Well, it doesn't, that's the point... unless I'm misunderstanding you here?
There was a problem hiding this comment.
We're all good (per my misunderstanding about PyDict_GetItem)
3ca7e5d to
ca68ed2
Compare
Accept passing a list of dicts to pa.array() if a struct type is given.
ca68ed2 to
2b9133a
Compare
|
+1, this looks good to go once the build runs. thanks @pitrou! |
|
(as a side note, who has the right to retry failed jobs on Travis? there are |
|
Folks with write access to the git repo (me, Uwe, @cpcloud, others) -- anyone on http://arrow.apache.org/committers/. I restarted the jobs. Not sure why apt on Travis has been so flaky lately |
|
Restarted the failed Travis CI build. Can you enable Appveyor on your fork so we can potentially get some faster builds to look at there? Thanks! |
|
Ok, there's an AppVeyor build starting at https://ci.appveyor.com/project/pitrou/arrow/build/1.0.2 |
Accept passing a list of dicts to pa.array() if a struct type is given.
Based on PR #1513.