You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Python] join_asof out-of-order error for big sorted tables #41706
Describe the usage question you have. Please include as many useful details as possible.
With pyarrow 16.0.0, I can't apply join_asof although the input tables are ordered by "on" key.
Noticed when trying to merge bigger sorted tables - for example, it fails for tables with rows numbers 1061753 & 994046, but can be executed if I reduce numbers to 1048178 & 975257.
I think this behavior can be reproduced with an example below:
So I suspect the issue has nothing to do with the on-key values order, but rather the input size?
Is it the bug that can be fixed or some fundamental limitation?
Is there any workaround other than limiting input size?
I was suspecting if pre-joining with by-key may cause the on-key reordering? As discussed here: https://lists.apache.org/list?user@arrow.apache.org:2024-4:join
However, the above example still fails even with empty by-key: t1.join_asof(t2, on="ts", tolerance=-10, by=[])
I think I narrowed down the problem. The as-of-join node does not sequence incoming ExecBatches. Here is a patch I created. It probably is not the optimal solution. Maybe sequencer should be incorporated in BackpressureConcurrentQueue?
changed the title [-]join_asof out-of-order error for big sorted tables[/-][+][C++] join_asof out-of-order error for big sorted tables[/+]on Nov 3, 2024
changed the title [-][C++] join_asof out-of-order error for big sorted tables[/-][+][Python] join_asof out-of-order error for big sorted tables[/+]on Nov 3, 2024
Activity
bepec commentedon May 17, 2024
I was suspecting if pre-joining with by-key may cause the on-key reordering? As discussed here: https://lists.apache.org/list?user@arrow.apache.org:2024-4:join
However, the above example still fails even with empty by-key:
t1.join_asof(t2, on="ts", tolerance=-10, by=[])0x26res commentedon May 23, 2024
I have a similar issue with a smaller table.
It only happens if I have a lot of small chunks in the table.
Here's an example:
it took a while to make a reproducible example. I can't exactly pin down what is causing the issue.
gitmodimo commentedon Jun 12, 2024
This also happened for me. I am using acero C++ api
gitmodimo commentedon Jun 20, 2024
I think I narrowed down the problem. The as-of-join node does not sequence incoming ExecBatches.
Here is a patch I created. It probably is not the optimal solution. Maybe sequencer should be incorporated in BackpressureConcurrentQueue?
mroz45 commentedon Sep 16, 2024
I created a PR with test that cause failure with specyfic parameters.
GH-41706: [C++][Acero] Enhance asof_join to work in multi-threaded ex…
westonpace commentedon Oct 29, 2024
Issue resolved by pull request 44083
#44083
[-]join_asof out-of-order error for big sorted tables[/-][+][C++] join_asof out-of-order error for big sorted tables[/+][-][C++] join_asof out-of-order error for big sorted tables[/-][+][Python] join_asof out-of-order error for big sorted tables[/+]GH-44629: [C++][Acero] Use `implicit_ordering` for `asof_join` rather…