Skip to content

ARROW-2033: [Python] Fix pa.array() with iterator input - #1513

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-2033-pa-array-iterator
Closed

pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-2033-pa-array-iterator

Conversation

@pitrou

@pitrou pitrou commented Jan 25, 2018

Copy link
Copy Markdown
Member

Iterator (not iterable) input was broken with pa.array() unless both type and size were explicitly passed.

@pitrou

pitrou commented Jan 29, 2018

Copy link
Copy Markdown
Member Author

There are (probably unrelated) build errors against the thrift header files:

In file included from src/parquet/column_reader.cc:34:0:
src/parquet/thrift.h: In function ‘void parquet::DeserializeThriftMsg(const uint8_t*, uint32_t*, T*)’:
src/parquet/thrift.h:103:48: error: no matching function for call to ‘apache::thrift::protocol::TCompactProtocolFactoryT<apache::thrift::transport::TMemoryBuffer>::getProtocol(std::shared_ptr<apache::thrift::transport::TMemoryBuffer>&)’
       tproto_factory.getProtocol(tmem_transport);
                                                ^
src/parquet/thrift.h:103:48: note: candidate is:
In file included from src/parquet/thrift.h:28:0,
                 from src/parquet/column_reader.cc:34:
/usr/include/thrift/protocol/TCompactProtocol.h:242:32: note: boost::shared_ptr<apache::thrift::protocol::TProtocol> apache::thrift::protocol::TCompactProtocolFactoryT<Transport_>::getProtocol(boost::shared_ptr<apache::thrift::transport::TTransport>) [with Transport_ = apache::thrift::transport::TMemoryBuffer]
   boost::shared_ptr<TProtocol> getProtocol(boost::shared_ptr<TTransport> trans) {
                                ^

How should I go about those?

@pitrou

pitrou commented Jan 29, 2018

Copy link
Copy Markdown
Member Author

Actually, PR #1528 is probably needed to get rid of the build errors.

@xhochy

xhochy commented Jan 29, 2018

Copy link
Copy Markdown
Member

@pitrou Ignore all Parquet or Thrift related errors for now, at the moment all of them boil down that they wait for conda-forge/thrift-cpp-feedstock#17

@pitrou

pitrou commented Jan 29, 2018

Copy link
Copy Markdown
Member Author

Ok, then I think this PR is basically ready.

@wesm

wesm commented Jan 29, 2018

Copy link
Copy Markdown
Member

Will review this today -- hope to get the PR backlog under control this week

@wesm wesm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made some minor stylistic comments but otherwise this looks good, thanks for refactoring / cleaning up this code path

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should have braces -- I would have thought that clang-format would add them but maybe not

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure how esoteric infinite iterators are in practice; at some point we might want some kind of "maximum size" to prevent hanging here

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NB. you can also indicate what std::min you want, e.g. std::min<int64_t>

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think seqref.ref() is only writable on account of const PyObject* being unwieldy -- it might be less awkward to pass a mutable pointer to seqref

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This size argument should precede the "out" arguments (https://google.github.io/styleguide/cppguide.html#Function_Parameter_Ordering)

Comment thread cpp/src/arrow/python/builtin_convert.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is a "can't fail" check; use DCHECK_GE

Comment thread cpp/src/arrow/python/builtin_convert.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: argument order

@wesm

wesm commented Jan 30, 2018

Copy link
Copy Markdown
Member

I'm going to try to fix these issues right now so we can get this merged, and then your other PR rebased tomorrow

@wesm

wesm commented Jan 30, 2018

Copy link
Copy Markdown
Member

@pitrou for future reference -- we prefer rebase to merge when updating from upstream, as it makes incremental PR maintenance a little bit easier

@wesm
wesm force-pushed the ARROW-2033-pa-array-iterator branch from c5314d1 to a0b146b Compare January 30, 2018 03:59
@wesm

wesm commented Jan 30, 2018

Copy link
Copy Markdown
Member

@xhochy @pitrou OK this is good to go once the build runs

@pitrou

pitrou commented Jan 30, 2018 •

Copy link
Copy Markdown
Member Author

@pitrou for future reference -- we prefer rebase to merge when updating from upstream, as it makes incremental PR maintenance a little bit easier

Hmm, that's weird. IME rebasing produces lots of spurious conflicts when trying to reconcile other changes.

For example trying to rebase or merge PR #1530 over the latest changes in this PR now produces lots of conflicts.

pitrou and others added 2 commits January 30, 2018 11:22
Change-Id: I76f3c31831c5faf1081dfb6ee3adf7498aca0997
Change-Id: I03ec91b8a90533b217fc1a6832a87b7722dca9d0
@pitrou
pitrou force-pushed the ARROW-2033-pa-array-iterator branch from ac8afdb to 0013889 Compare January 30, 2018 10:22
@pitrou

pitrou commented Jan 30, 2018

Copy link
Copy Markdown
Member Author

Unrelated, but FTR, the Travis-CI connectivity issues have an issue open at travis-ci/travis-ci#9112

@xhochy

xhochy commented Jan 30, 2018

Copy link
Copy Markdown
Member

@pitrou Thank you for the reference to the Travis issue. I've retrigged all tests that were failed because of that, we're still waiting for AppVeyor. Once that is done, I'll merge.

@pitrou

pitrou commented Jan 30, 2018 •

Copy link
Copy Markdown
Member Author

One Travis-CI job has failed with pyarrow.lib.ArrowIOError: Could not connect to socket /tmp/plasma_store47336278. Also that job was unexpectedly slow:
https://travis-ci.org/apache/arrow/jobs/335088022

@wesm

wesm commented Jan 30, 2018

Copy link
Copy Markdown
Member

That was https://issues.apache.org/jira/browse/ARROW-1880 striking again. We haven't figured out yet what causes that, but it seems to be the Travis CI VMs having limited resources

@wesm wesm closed this Jan 30, 2018
@pitrou
pitrou deleted the ARROW-2033-pa-array-iterator branch January 30, 2018 15:18
wesm pushed a commit that referenced this pull request Jan 30, 2018
Iterator (not iterable) input was broken with pa.array() unless both type and size were explicitly passed.

Author: Wes McKinney <wes.mckinney@twosigma.com>
Author: Antoine Pitrou <pitrou@free.fr>

Closes #1513 from pitrou/ARROW-2033-pa-array-iterator and squashes the following commits:

0013889 [Wes McKinney] Code review comments
dc95be2 [Antoine Pitrou] Fix pyarrow.array with iterator input

Change-Id: I930c3309e3fde12e65ede066b47985f67f7f4037
wesm pushed a commit that referenced this pull request Jan 30, 2018
Accept passing a list of dicts to pa.array() if a struct type is given.

Based on PR #1513.

Author: Antoine Pitrou <antoine@python.org>

Closes #1530 from pitrou/ARROW-1705-struct-array-from-dicts and squashes the following commits:

2b9133a [Antoine Pitrou] ARROW-1705: [Python] allow building array from dicts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants