Repository navigation
Conversation
|
Not sure why getting in CI the error That function has all arguments documented, will need to dive a bit more into why it's complaining. |
jorisvandenbossche
left a comment
There was a problem hiding this comment.
I did a first quick round through the code and added some minor inline comments.
I also tried it out and have some user comments:
- Currently, the "key" column (column on which is joined) is included twice in the output (from both left and right table). I don't know if this is inherent to how the Join node of the exec plan works, but I would expect that the
joinmethod will handle such de-duplication for me (I suppose on the R side this also done, so might be worth checking out how they handle that) - You also need to specify the key column for both left and right table separate. While this is certainly the most generic (since it can handle different names in left and right table), I think it could also be nice to give the user the possibility to just give one name (or list of names) in case it is the same in left/right table (for better ergonomics when using this method)
- In pandas we provide the option to add suffixes to the left/right table names in case there are clashes. It seems that the C++ join options have some feature for that as well (but that's something that is certainly not essential for a first PR)
There was a problem hiding this comment.
We already have the declaration for GetCpuThreadPoolCapacity from thread_pool.h somewhere in this file, maybe move them together?
There was a problem hiding this comment.
They have different namespaces by the way, the GetCpuThreadPoolCapacity is arrow:: while the CExecutor is arrow:internal::
There was a problem hiding this comment.
They have different namespaces by the way, the
GetCpuThreadPoolCapacityisarrow::while theCExecutorisarrow:internal::
That's more a C++ question, but if we use this here, shouldn't that ideally be moved in the arrow namespace instead of the internal?(cc @pitrou)
There was a problem hiding this comment.
Longer term, I think the question is to what extent we want to factor out this execplan function in a "nicer" interface (even for cython), eg class based, or .. But I suppose for this PR itself it is good enough.
There was a problem hiding this comment.
For now the plan is to avoid exposing execplan to end user, so the interface is low level.
We should discuss in the near future about what we want to expose given that I think it would be reasonable to have an interface to lazily evaluate expressions (compared to doing table.join().group_by() which evaluates them step by step). But that shouldn't probably expose execplan itself, it should instead build one.
|
@github-actions crossbow submit -g test |
|
Revision: a9155f9fbbac2166894a69348347e5767ac48f36 Submitted crossbow builds: ursacomputing/crossbow @ actions-1716 |
|
Added |
|
There seem to be a bunch of problems with CI unrelated to the PR, there are failures related to and |
@jorisvandenbossche mind confirming if the PR is now good enough to go? I'd like to start working on support for Datasets and docs. |
|
@amol- Can you rebase? |
|
I would personally prefer to see this comment addressed as well (or at least get some thoughts on it):
For the join keys columns in the output: you now selected one of the columns for most joins, but not for outer join, I think? I am not fully sure if we should do something different here for outer join (for example, both pandas and dplyr will only have a single key column in the output also in the case of an outer join) |
|
IMHO, it would be more consistent to get two key columns for all joins. |
I'll add support for omitting the right table keys and suffixing columns in the output as supported by HashJoinNodeOptions.
That's an interesting point. Personally I think that for outer joins it makes a lot of sense to have both columns. Coalescing the key columns would make the information about from which table the key comes from getting lost. I think it's more reasonable to let users decide if they want to coalesce outer join keys or not, especially given that the coalesce operation would add a cost as we don't provide it in joins out of the box. For example VS |
rebased! |
|
Thanks for the updates!
It indeed gives more information (although you will also often see it from other columns having all nulls if the key was not present). But there is also the expectation / usability for people coming from libraries like pandas or dplyr. Now, I was looking a bit at some other libraries how they handle this. So pandas and dplyr combine the both join columns in a single column. I was checking ibis, but I think it might depend on the backend you are using in that case. You need to actively select the columns you want to retain after the join (so you could say that by default they retain both join columns). With the pandas backend though, it does coalesce nulls in the left key column with values from the right column. But eg the sqlite backend does not do that for a "right" join (this backend doesn't support an outer join). Duckdb seems to preserve both left/right join columns for inner/left join (and doesn't support an outer join). In general, I think the behaviour depends on whether the library is SQL "based" (wrapping sql, or using mostly sql concepts) or not (where dubck and ibis fall into that group). Since in SQL, you always have to explicitly select columns as well (a project) while doing a join. Some of my explorations: https://notebooksharing.space/view/902129774575d83b4e3096d4a7c1eb26b8e830842096d3dc01a600d220848d99#displayOptions=
That's certainly the most consistent, but also not always very practical IMO. For example, we can't easily deal with duplicate column names. As an example, if you would do a |
|
Given that Arrow is a bit in between (or has users from both) the database and dataframe worlds, I am wondering if we can provide both expected results through some keyword? |
jorisvandenbossche
left a comment
There was a problem hiding this comment.
Thanks a lot for the additional work on this! This makes the implementation more complex, but I think this has high value for the user.
There was a problem hiding this comment.
| c_executor_sptr = GetResultValue(CThreadPool.Make(1)) | |
| c_executor = c_executor_sptr.get() | |
| c_executor = GetResultValue(CThreadPool.Make(1)).get() |
Cython doesn't understand it when doing in one go? (or use .get() in the line below where c_executor is passed to create the context)
There was a problem hiding this comment.
We might want to expose this keyword here as well and pass it through, so that users can set it to False if they want?
There was a problem hiding this comment.
And to bike-shed about the name: since the columns are not actually identical (they have complementary information which gets coalesced), the "duplicate" might be a bit misleading. Something like "combine_key_columns=True/False" might be more descriptive, although a bit long?
dplyr has this as the "keep" argument (https://dplyr.tidyverse.org/reference/mutate-joins.html), so that's another option as prior art (although I don't find that a very descriptive name ..)
There was a problem hiding this comment.
coalesce_keys=True/False seems a reasonable name that is not too long.
There was a problem hiding this comment.
coalesce_keys sounds good, thanks!
jorisvandenbossche
left a comment
There was a problem hiding this comment.
Thanks for the final updates!
You will need to do one more rebase, there is a conflict because of some unrelated changes in io.pxi
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
|
Benchmark runs are scheduled for baseline = 4d0436a and contender = 09a794a. 09a794a is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
This implements the
Table.joinmethod and the underlying infrastructure.It provides the
tables_joinfunction that wraps theexecplanmachinery and tests to verify that joins work as expected.