Skip to content

ARROW-14292: [C++][Python] Join foundation for Tables - #12452

Closed
amol- wants to merge 24 commits into
apache:masterfrom
amol-:ARROW-14292
Closed

amol- wants to merge 24 commits into
apache:masterfrom
amol-:ARROW-14292

Conversation

@amol-

@amol- amol- commented Feb 17, 2022 •

Copy link
Copy Markdown
Member

This implements the Table.join method and the underlying infrastructure.
It provides the tables_join function that wraps the execplan machinery and tests to verify that joins work as expected.

@github-actions

Copy link
Copy Markdown

@amol-
amol- marked this pull request as ready for review February 18, 2022 15:33
@amol-

amol- commented Feb 18, 2022

Copy link
Copy Markdown
Member Author

Not sure why getting in CI the error

pyarrow._exec_plan.tables_join
-> pyarrow._exec_plan.tables_join(join_type, left_table, left_keys, right_table, right_keys)
PR01: Parameters {'left_keys', 'left_table', 'right_table', 'right_keys'} not documented

That function has all arguments documented, will need to dive a bit more into why it's complaining.

@jorisvandenbossche jorisvandenbossche left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a first quick round through the code and added some minor inline comments.

I also tried it out and have some user comments:

  • Currently, the "key" column (column on which is joined) is included twice in the output (from both left and right table). I don't know if this is inherent to how the Join node of the exec plan works, but I would expect that the join method will handle such de-duplication for me (I suppose on the R side this also done, so might be worth checking out how they handle that)
  • You also need to specify the key column for both left and right table separate. While this is certainly the most generic (since it can handle different names in left and right table), I think it could also be nice to give the user the possibility to just give one name (or list of names) in case it is the same in left/right table (for better ergonomics when using this method)
  • In pandas we provide the option to add suffixes to the left/right table names in case there are clashes. It seems that the C++ join options have some feature for that as well (but that's something that is certainly not essential for a first PR)

Comment thread python/pyarrow/_exec_plan.pyx Resolved
Comment thread python/pyarrow/_exec_plan.pyx Resolved
Comment thread python/pyarrow/includes/libarrow.pxd Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have the declaration for GetCpuThreadPoolCapacity from thread_pool.h somewhere in this file, maybe move them together?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They have different namespaces by the way, the GetCpuThreadPoolCapacity is arrow:: while the CExecutor is arrow:internal::

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They have different namespaces by the way, the GetCpuThreadPoolCapacity is arrow:: while the CExecutor is arrow:internal::

That's more a C++ question, but if we use this here, shouldn't that ideally be moved in the arrow namespace instead of the internal?(cc @pitrou)

Comment thread python/pyarrow/_exec_plan.pyx Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Longer term, I think the question is to what extent we want to factor out this execplan function in a "nicer" interface (even for cython), eg class based, or .. But I suppose for this PR itself it is good enough.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now the plan is to avoid exposing execplan to end user, so the interface is low level.

We should discuss in the near future about what we want to expose given that I think it would be reasonable to have an interface to lazily evaluate expressions (compared to doing table.join().group_by() which evaluates them step by step). But that shouldn't probably expose execplan itself, it should instead build one.

Comment thread python/pyarrow/_exec_plan.pyx Resolved
Comment thread python/pyarrow/_exec_plan.pyx Resolved
Comment thread python/pyarrow/_exec_plan.pyx Resolved
Comment thread python/pyarrow/_exec_plan.pyx Resolved
Comment thread python/pyarrow/_exec_plan.pyx Resolved
@amol-

amol- commented Mar 3, 2022 •

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g test

@github-actions

github-actions Bot commented Mar 3, 2022

Copy link
Copy Markdown

Revision: a9155f9fbbac2166894a69348347e5767ac48f36

Submitted crossbow builds: ursacomputing/crossbow @ actions-1716

Task Status
test-build-cpp-fuzz Github Actions
test-build-vcpkg-win Github Actions
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.7-turbodbc-latest Github Actions
test-conda-python-3.7-turbodbc-master Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions
test-debian-10-cpp-amd64 Github Actions
test-debian-10-cpp-i386 Github Actions
test-debian-11-cpp-amd64 Github Actions
test-debian-11-cpp-i386 Github Actions
test-debian-11-go-1.16 Azure
test-debian-11-python-3 Azure
test-debian-c-glib Github Actions
test-debian-ruby Github Actions
test-fedora-33-cpp Github Actions
test-fedora-33-python-3 Azure
test-fedora-r-clang-sanitizer Azure
test-r-arrow-backwards-compatibility Github Actions
test-r-depsource-bundled Azure
test-r-depsource-system Github Actions
test-r-dev-duckdb Github Actions
test-r-devdocs Github Actions
test-r-gcc-11 Github Actions
test-r-install-local Github Actions
test-r-linux-as-cran Github Actions
test-r-linux-rchk Github Actions
test-r-linux-valgrind Azure
test-r-minimal-build Azure
test-r-offline-maximal Github Actions
test-r-offline-minimal Azure
test-r-rhub-debian-gcc-devel-lto-latest Azure
test-r-rhub-ubuntu-gcc-release-latest Azure
test-r-rocker-r-base-latest Azure
test-r-rstudio-r-base-4.1-centos7-devtoolset-8 Azure
test-r-rstudio-r-base-4.1-focal Azure
test-r-rstudio-r-base-4.1-opensuse15 Azure
test-r-rstudio-r-base-4.1-opensuse42 Azure
test-r-ubuntu-21.04 Github Actions
test-r-versions Github Actions
test-skyhook-integration Github Actions
test-ubuntu-18.04-cpp Github Actions
test-ubuntu-18.04-cpp-release Github Actions
test-ubuntu-18.04-cpp-static Github Actions
test-ubuntu-18.04-r-sanitizer Azure
test-ubuntu-20.04-cpp Github Actions
test-ubuntu-20.04-cpp-14 Github Actions
test-ubuntu-20.04-cpp-17 Github Actions
test-ubuntu-20.04-cpp-bundled Github Actions
test-ubuntu-20.04-cpp-thread-sanitizer Github Actions
test-ubuntu-20.04-python-3 Azure
test-ubuntu-21.04-cpp Github Actions
test-ubuntu-c-glib Github Actions
test-ubuntu-default-docs Azure
test-ubuntu-ruby Github Actions

@apache apache deleted a comment from github-actions Bot Mar 3, 2022
@amol-

amol- commented Mar 4, 2022

Copy link
Copy Markdown
Member Author

@jorisvandenbossche

Added use_threads option and added support for excluding the left/right keys on specific joins.

@amol-

amol- commented Mar 4, 2022

Copy link
Copy Markdown
Member Author

There seem to be a bunch of problems with CI unrelated to the PR, there are failures related to

Error: getCacheEntry failed: connect ETIMEDOUT 13.107.42.16:443

and

CMake Error at /opt/conda/envs/arrow/lib/cmake/AWSSDK/AWSSDKConfig.cmake:119 (while):
  had incorrect arguments:

    "NOT" "TEMP_NAME" "STREQUAL"

  Unknown arguments specified

@amol-

amol- commented Mar 9, 2022

Copy link
Copy Markdown
Member Author

@jorisvandenbossche

Added use_threads option and added support for excluding the left/right keys on specific joins.

@jorisvandenbossche mind confirming if the PR is now good enough to go? I'd like to start working on support for Datasets and docs.

@pitrou

pitrou commented Mar 10, 2022

Copy link
Copy Markdown
Member

@amol- Can you rebase?

@jorisvandenbossche

Copy link
Copy Markdown
Member

I would personally prefer to see this comment addressed as well (or at least get some thoughts on it):

You also need to specify the key column for both left and right table separate. While this is certainly the most generic (since it can handle different names in left and right table), I think it could also be nice to give the user the possibility to just give one name (or list of names) in case it is the same in left/right table (for better ergonomics when using this method)

For the join keys columns in the output: you now selected one of the columns for most joins, but not for outer join, I think? I am not fully sure if we should do something different here for outer join (for example, both pandas and dplyr will only have a single key column in the output also in the case of an outer join)

@pitrou

pitrou commented Mar 10, 2022

Copy link
Copy Markdown
Member

IMHO, it would be more consistent to get two key columns for all joins.

@amol-

amol- commented Mar 11, 2022 •

Copy link
Copy Markdown
Member Author

I would personally prefer to see this comment addressed as well (or at least get some thoughts on it):

You also need to specify the key column for both left and right table separate. While this is certainly the most generic (since it can handle different names in left and right table), I think it could also be nice to give the user the possibility to just give one name (or list of names) in case it is the same in left/right table (for better ergonomics when using this method)

I'll add support for omitting the right table keys and suffixing columns in the output as supported by HashJoinNodeOptions.

For the join keys columns in the output: you now selected one of the columns for most joins, but not for outer join, I think? I am not fully sure if we should do something different here for outer join (for example, both pandas and dplyr will only have a single key column in the output also in the case of an outer join)

That's an interesting point. Personally I think that for outer joins it makes a lot of sense to have both columns. Coalescing the key columns would make the information about from which table the key comes from getting lost. I think it's more reasonable to let users decide if they want to coalesce outer join keys or not, especially given that the coalesce operation would add a cost as we don't provide it in joins out of the box.

For example

Key | Key_t2 | Other
 1  |  null  |  55            <---- Obvious that the value comes from Table1 and not Table2

VS

Key | Other
 1  |  55                     <---- Where did "1" come from?

@amol-

amol- commented Mar 11, 2022

Copy link
Copy Markdown
Member Author

@amol- Can you rebase?

rebased!

@jorisvandenbossche

Copy link
Copy Markdown
Member

Thanks for the updates!
(small workflow request: could you keep new changes in a new commit instead of squashing it into a previous one? That makes it much easier to see what you changed since the last review)

That's an interesting point. Personally I think that for outer joins it makes a lot of sense to have both columns. Coalescing the key columns would make the information about from which table the key comes from getting lost. I think it's more reasonable to let users decide if they want to coalesce outer join keys or not, especially given that the coalesce operation would add a cost as we don't provide it in joins out of the box.

It indeed gives more information (although you will also often see it from other columns having all nulls if the key was not present). But there is also the expectation / usability for people coming from libraries like pandas or dplyr.

Now, I was looking a bit at some other libraries how they handle this. So pandas and dplyr combine the both join columns in a single column. I was checking ibis, but I think it might depend on the backend you are using in that case. You need to actively select the columns you want to retain after the join (so you could say that by default they retain both join columns). With the pandas backend though, it does coalesce nulls in the left key column with values from the right column. But eg the sqlite backend does not do that for a "right" join (this backend doesn't support an outer join).

Duckdb seems to preserve both left/right join columns for inner/left join (and doesn't support an outer join).
Polars follows the pandas / dplyr behaviour.

In general, I think the behaviour depends on whether the library is SQL "based" (wrapping sql, or using mostly sql concepts) or not (where dubck and ibis fall into that group). Since in SQL, you always have to explicitly select columns as well (a project) while doing a join.

Some of my explorations: https://notebooksharing.space/view/902129774575d83b4e3096d4a7c1eb26b8e830842096d3dc01a600d220848d99#displayOptions=

IMHO, it would be more consistent to get two key columns for all joins.

That's certainly the most consistent, but also not always very practical IMO. For example, we can't easily deal with duplicate column names. As an example, if you would do a group_by after the join, you get errors about duplicate columns names (then you can of course use the suffix option, but that's also not super convenient).

Comment thread python/pyarrow/table.pxi Resolved
@jorisvandenbossche

Copy link
Copy Markdown
Member

Given that Arrow is a bit in between (or has users from both) the database and dataframe worlds, I am wondering if we can provide both expected results through some keyword?
The keyword could determine whether we return a single join column (and in that case I think we could also "combine" / coalesce them automatically) or preserve both columns as is in the output.

@jorisvandenbossche jorisvandenbossche left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the additional work on this! This makes the implementation more complex, but I think this has high value for the user.

Comment thread python/pyarrow/_compute.pyx Resolved
Comment thread python/pyarrow/_exec_plan.pyx Outdated
Comment on lines 72 to 73

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
c_executor_sptr = GetResultValue(CThreadPool.Make(1))
c_executor = c_executor_sptr.get()
c_executor = GetResultValue(CThreadPool.Make(1)).get()

Cython doesn't understand it when doing in one go? (or use .get() in the line below where c_executor is passed to create the context)

Comment thread python/pyarrow/table.pxi Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might want to expose this keyword here as well and pass it through, so that users can set it to False if they want?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And to bike-shed about the name: since the columns are not actually identical (they have complementary information which gets coalesced), the "duplicate" might be a bit misleading. Something like "combine_key_columns=True/False" might be more descriptive, although a bit long?

dplyr has this as the "keep" argument (https://dplyr.tidyverse.org/reference/mutate-joins.html), so that's another option as prior art (although I don't find that a very descriptive name ..)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

coalesce_keys=True/False seems a reasonable name that is not too long.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

coalesce_keys sounds good, thanks!

Comment thread python/pyarrow/_exec_plan.pyx Resolved
Comment thread python/pyarrow/_exec_plan.pyx Resolved
Comment thread python/pyarrow/_exec_plan.pyx Resolved
Comment thread python/pyarrow/_exec_plan.pyx Resolved

@jorisvandenbossche jorisvandenbossche left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the final updates!

You will need to do one more rebase, there is a conflict because of some unrelated changes in io.pxi

@ursabot

ursabot commented Mar 29, 2022 •

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 4d0436a and contender = 09a794a. 09a794a is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.88% ⬆️0.0%] test-mac-arm
[Finished ⬇️1.07% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.47% ⬆️0.04%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/408| 09a794a6 ec2-t3-xlarge-us-east-2>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/394| 09a794a6 test-mac-arm>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/394| 09a794a6 ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/404| 09a794a6 ursa-thinkcentre-m75q>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/407| 4d0436a0 ec2-t3-xlarge-us-east-2>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/393| 4d0436a0 test-mac-arm>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/393| 4d0436a0 ursa-i9-9960x>
[Finished] <https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/403| 4d0436a0 ursa-thinkcentre-m75q>
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants