Skip to content

GH-44629: [C++][Acero] Use implicit_ordering for asof_join rather than require_sequenced_output - #44616

Merged
zanmato1984 merged 3 commits into
apache:mainfrom
EnricoMi:asof-join-followup
Feb 11, 2025
Merged

zanmato1984 merged 3 commits into
apache:mainfrom
EnricoMi:asof-join-followup

Conversation

@EnricoMi

@EnricoMi EnricoMi commented Nov 1, 2024 •

Copy link
Copy Markdown
Collaborator

Rationale for this change

Changes in #44083 (GH-41706) unnecessarily sequences batches retrieved from scanner where it only requires the batches to provide index according to implicit input order.

What changes are included in this PR?

Setting implicit_ordering causes existing code to set batch index, which is then available to the asof_join node to sequence the batches int input order. This replaces some of #44083 changes.

Some code introduced by #44083 turns out to not be required and has therefore been reverted.

Are these changes tested?

Existing unit tests still pass.

Are there any user-facing changes?

Reverts some breaking user-facing changes done by #44083.

@gitmodimo gitmodimo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good. I think all source nodes need an option to assert Implicit ordering but I am not sure if it is in the scope of this PR.
Is there currently any use for require_sequenced_output?

@EnricoMi

EnricoMi commented Nov 1, 2024

Copy link
Copy Markdown
Collaborator Author

@westonpace are you happy with this?

@kou

kou commented Nov 2, 2024

Copy link
Copy Markdown
Member

Could you open a new issue instead of reusing closed issue?

@EnricoMi EnricoMi changed the title GH-41706: [C++][Acero][Followup] Use implicit_ordering for asof_join rather than require_sequenced_output GH-44629: [C++][Acero] Use implicit_ordering for asof_join rather than require_sequenced_output Nov 3, 2024
@github-actions

github-actions Bot commented Nov 3, 2024

Copy link
Copy Markdown

⚠️ GitHub issue #44629 has been automatically assigned in GitHub to PR creator.

@EnricoMi

EnricoMi commented Nov 3, 2024

Copy link
Copy Markdown
Collaborator Author

@kou done

@rok
rok requested a review from felipecrv November 7, 2024 12:15
@EnricoMi

Copy link
Copy Markdown
Collaborator Author

@lidavidm what do you think?

@rok
rok requested a review from gitmodimo November 15, 2024 11:34

@gitmodimo gitmodimo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if it in the scope of this PR, but Implicit Ordering in other source nodes also need some consideration. My thoughts:
SourceNode - ordering depends on generator and can be asserted by option - seems reasonable
TableSourceNode - currently has implicit ordering which seems reasonable since tables do have an order. Probably little to none benefit in dropping the it.
SchemaSourceNode,RecordBatchSourceNode - currently has implicit ordering, but ordering depends on generator/iterator and should not default to either. I think it should be on option like in SourceNode
RecordBatchReaderSourceNode - currently has no implicit ordering - Same case as SchemaSourceNode

I think either all source nodes should have ordering option or the ordering should originate from arrow::AsyncGenerator ?

Other than that it looks good. Thanks for fixing my mistakes.

@lidavidm

Copy link
Copy Markdown
Member

Sorry @EnricoMi I wasn't involved in most of the Acero development so I can't really speak to the workings here. @bkietz would probably know better

@EnricoMi

EnricoMi commented Dec 4, 2024

Copy link
Copy Markdown
Collaborator Author

@westonpace since you approved #44083, can you take a look at this fix?

@EnricoMi

Copy link
Copy Markdown
Collaborator Author

@zanmato1984 can you have a look at this cleanup of an earlier PR, please?

@zanmato1984

Copy link
Copy Markdown
Contributor

OK, I'll take a look.

@zanmato1984 zanmato1984 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM. Just several questions. Thank you.

Comment thread cpp/src/arrow/dataset/scanner.h Resolved
Comment thread cpp/src/arrow/dataset/scanner.h Resolved
Comment thread python/pyarrow/_dataset.pyx Resolved

@zanmato1984 zanmato1984 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@zanmato1984

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g cpp -g python

@github-actions

Copy link
Copy Markdown

Revision: a3c8ff074f46d93abcdf1ca0f13b2c43026e74d4

Submitted crossbow builds: ursacomputing/crossbow @ actions-d5e23273f6

Task Status
example-cpp-minimal-build-static GitHub Actions
example-cpp-minimal-build-static-system-dependency GitHub Actions
example-cpp-tutorial GitHub Actions
example-python-minimal-build-fedora-conda GitHub Actions
example-python-minimal-build-ubuntu-venv GitHub Actions
test-alpine-linux-cpp GitHub Actions
test-build-cpp-fuzz GitHub Actions
test-conda-cpp GitHub Actions
test-conda-cpp-valgrind GitHub Actions
test-conda-python-3.10 GitHub Actions
test-conda-python-3.10-hdfs-2.9.2 GitHub Actions
test-conda-python-3.10-hdfs-3.2.1 GitHub Actions
test-conda-python-3.10-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.10-substrait GitHub Actions
test-conda-python-3.11 GitHub Actions
test-conda-python-3.11-dask-latest GitHub Actions
test-conda-python-3.11-dask-upstream_devel GitHub Actions
test-conda-python-3.11-hypothesis GitHub Actions
test-conda-python-3.11-pandas-latest-numpy-1.26 GitHub Actions
test-conda-python-3.11-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.11-pandas-nightly-numpy-nightly GitHub Actions
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly GitHub Actions
test-conda-python-3.11-spark-master GitHub Actions
test-conda-python-3.12 GitHub Actions
test-conda-python-3.12-cpython-debug GitHub Actions
test-conda-python-3.13 GitHub Actions
test-conda-python-3.9 GitHub Actions
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5 GitHub Actions
test-conda-python-emscripten GitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2 GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1 GitHub Actions
test-cuda-python-ubuntu-22.04-cuda-11.7.1 GitHub Actions
test-debian-12-cpp-amd64 GitHub Actions
test-debian-12-cpp-i386 GitHub Actions
test-debian-12-python-3-amd64 GitHub Actions
test-debian-12-python-3-i386 GitHub Actions
test-fedora-39-cpp GitHub Actions
test-fedora-39-python-3 GitHub Actions
test-ubuntu-20.04-cpp GitHub Actions
test-ubuntu-20.04-cpp-bundled GitHub Actions
test-ubuntu-22.04-cpp GitHub Actions
test-ubuntu-22.04-cpp-20 GitHub Actions
test-ubuntu-22.04-cpp-emscripten GitHub Actions
test-ubuntu-22.04-cpp-no-threading GitHub Actions
test-ubuntu-22.04-python-3 GitHub Actions
test-ubuntu-22.04-python-313-freethreading GitHub Actions
test-ubuntu-24.04-cpp GitHub Actions
test-ubuntu-24.04-cpp-bundled-offline GitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundled GitHub Actions
test-ubuntu-24.04-cpp-gcc-14 GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formats GitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizer GitHub Actions
test-ubuntu-24.04-python-3 GitHub Actions

@zanmato1984

Copy link
Copy Markdown
Contributor

I saw some CI failures so I performed a rebase. Triggering CI again.

@zanmato1984

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g cpp -g python

@github-actions

Copy link
Copy Markdown

Revision: 8af3ba9

Submitted crossbow builds: ursacomputing/crossbow @ actions-a3836be315

Task Status
example-cpp-minimal-build-static GitHub Actions
example-cpp-minimal-build-static-system-dependency GitHub Actions
example-cpp-tutorial GitHub Actions
example-python-minimal-build-fedora-conda GitHub Actions
example-python-minimal-build-ubuntu-venv GitHub Actions
test-alpine-linux-cpp GitHub Actions
test-build-cpp-fuzz GitHub Actions
test-conda-cpp GitHub Actions
test-conda-cpp-valgrind GitHub Actions
test-conda-python-3.10 GitHub Actions
test-conda-python-3.10-hdfs-2.9.2 GitHub Actions
test-conda-python-3.10-hdfs-3.2.1 GitHub Actions
test-conda-python-3.10-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.10-substrait GitHub Actions
test-conda-python-3.11 GitHub Actions
test-conda-python-3.11-dask-latest GitHub Actions
test-conda-python-3.11-dask-upstream_devel GitHub Actions
test-conda-python-3.11-hypothesis GitHub Actions
test-conda-python-3.11-pandas-latest-numpy-1.26 GitHub Actions
test-conda-python-3.11-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.11-pandas-nightly-numpy-nightly GitHub Actions
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly GitHub Actions
test-conda-python-3.11-spark-master GitHub Actions
test-conda-python-3.12 GitHub Actions
test-conda-python-3.12-cpython-debug GitHub Actions
test-conda-python-3.13 GitHub Actions
test-conda-python-3.9 GitHub Actions
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5 GitHub Actions
test-conda-python-emscripten GitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2 GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1 GitHub Actions
test-cuda-python-ubuntu-22.04-cuda-11.7.1 GitHub Actions
test-debian-12-cpp-amd64 GitHub Actions
test-debian-12-cpp-i386 GitHub Actions
test-debian-12-python-3-amd64 GitHub Actions
test-debian-12-python-3-i386 GitHub Actions
test-fedora-39-cpp GitHub Actions
test-fedora-39-python-3 GitHub Actions
test-ubuntu-20.04-cpp GitHub Actions
test-ubuntu-20.04-cpp-bundled GitHub Actions
test-ubuntu-22.04-cpp GitHub Actions
test-ubuntu-22.04-cpp-20 GitHub Actions
test-ubuntu-22.04-cpp-emscripten GitHub Actions
test-ubuntu-22.04-cpp-no-threading GitHub Actions
test-ubuntu-22.04-python-3 GitHub Actions
test-ubuntu-22.04-python-313-freethreading GitHub Actions
test-ubuntu-24.04-cpp GitHub Actions
test-ubuntu-24.04-cpp-bundled-offline GitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundled GitHub Actions
test-ubuntu-24.04-cpp-gcc-14 GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formats GitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizer GitHub Actions
test-ubuntu-24.04-python-3 GitHub Actions

@zanmato1984

Copy link
Copy Markdown
Contributor

The CI failures seem unrelated, they have been failing in nightly for days.

I'm merging this. Thank you @EnricoMi for this!

@EnricoMi

Copy link
Copy Markdown
Collaborator Author

@zanmato1984 thank you very much for getting this fix in!

@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 9410327.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants