Repository navigation
GH-44629: [C++][Acero] Use implicit_ordering for asof_join rather than require_sequenced_output - #44616
Conversation
gitmodimo
left a comment
There was a problem hiding this comment.
All looks good. I think all source nodes need an option to assert Implicit ordering but I am not sure if it is in the scope of this PR.
Is there currently any use for require_sequenced_output?
|
@westonpace are you happy with this? |
|
Could you open a new issue instead of reusing closed issue? |
implicit_ordering for asof_join rather than require_sequenced_outputimplicit_ordering for asof_join rather than require_sequenced_output
|
|
|
@kou done |
|
@lidavidm what do you think? |
gitmodimo
left a comment
There was a problem hiding this comment.
I don't know if it in the scope of this PR, but Implicit Ordering in other source nodes also need some consideration. My thoughts:
SourceNode - ordering depends on generator and can be asserted by option - seems reasonable
TableSourceNode - currently has implicit ordering which seems reasonable since tables do have an order. Probably little to none benefit in dropping the it.
SchemaSourceNode,RecordBatchSourceNode - currently has implicit ordering, but ordering depends on generator/iterator and should not default to either. I think it should be on option like in SourceNode
RecordBatchReaderSourceNode - currently has no implicit ordering - Same case as SchemaSourceNode
I think either all source nodes should have ordering option or the ordering should originate from arrow::AsyncGenerator ?
Other than that it looks good. Thanks for fixing my mistakes.
|
@westonpace since you approved #44083, can you take a look at this fix? |
|
@zanmato1984 can you have a look at this cleanup of an earlier PR, please? |
b436a8a to
afb0783
Compare
|
OK, I'll take a look. |
zanmato1984
left a comment
There was a problem hiding this comment.
This LGTM. Just several questions. Thank you.
afb0783 to
b381181
Compare
|
@github-actions crossbow submit -g cpp -g python |
|
Revision: a3c8ff074f46d93abcdf1ca0f13b2c43026e74d4 Submitted crossbow builds: ursacomputing/crossbow @ actions-d5e23273f6 |
a3c8ff0 to
8af3ba9
Compare
|
I saw some CI failures so I performed a rebase. Triggering CI again. |
|
@github-actions crossbow submit -g cpp -g python |
|
Revision: 8af3ba9 Submitted crossbow builds: ursacomputing/crossbow @ actions-a3836be315 |
|
The CI failures seem unrelated, they have been failing in nightly for days. I'm merging this. Thank you @EnricoMi for this! |
|
@zanmato1984 thank you very much for getting this fix in! |
|
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 9410327. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. |
Rationale for this change
Changes in #44083 (GH-41706) unnecessarily sequences batches retrieved from scanner where it only requires the batches to provide index according to implicit input order.
What changes are included in this PR?
Setting
implicit_orderingcauses existing code to set batch index, which is then available to theasof_joinnode to sequence the batches int input order. This replaces some of #44083 changes.Some code introduced by #44083 turns out to not be required and has therefore been reverted.
Are these changes tested?
Existing unit tests still pass.
Are there any user-facing changes?
Reverts some breaking user-facing changes done by #44083.
implicit_orderingforasof_joinrather thanrequire_sequenced_output#44629