Skip to content

GH-34698: [C++][Python][Acero] Add node that emits explicit ordering after asserting order - #44738

Closed
EnricoMi wants to merge 14 commits into
apache:mainfrom
EnricoMi:assert-order-node
Closed

EnricoMi wants to merge 14 commits into
apache:mainfrom
EnricoMi:assert-order-node

Conversation

@EnricoMi

@EnricoMi EnricoMi commented Nov 15, 2024 •

Copy link
Copy Markdown
Collaborator

Rationale for this change

An acero node that turns an implicit ordering into an explicit ordering (rows sorted by some columns) is useful to re-use order that already exists in the data.

What changes are included in this PR?

This PR adds the AssertOrderNode that implements this logic. The Scanner employs that node to turn the implicit ordering of the ScanNode into an explicit order as defined by user code via ScanBuilder.Ordering.

Are these changes tested?

There are unit tests for the AssertOrderNode as well as for the ScanNode and ScanBuilder.

Are there any user-facing changes?

The following options has been added to C++:

  • the ordering option added to ScanOptions
  • the Ordering option added to ScannerBuilder
  • the AssertNodeOptions class

The following method is added to Python Dataset:

  • method assert_sorted asserts given order

@EnricoMi
EnricoMi requested a review from westonpace as a code owner November 15, 2024 09:29
@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

@EnricoMi EnricoMi changed the title GH-#34698: [C++] Add node that emits explicit ordering after asserting order GH-#34698: [C++][Acero] Add node that emits explicit ordering after asserting order Nov 15, 2024
@EnricoMi EnricoMi changed the title GH-#34698: [C++][Acero] Add node that emits explicit ordering after asserting order GH-34698: [C++][Acero] Add node that emits explicit ordering after asserting order Nov 15, 2024
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #34698 has been automatically assigned in GitHub to PR creator.

@raulcd raulcd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am probably not qualified to review this thoroughly but the CI failure is related:


[  FAILED  ] TestScannerThreading/TestScanner.ScanOrderingFail/0Serial1d1b1024r, where GetParam() = serial-1d-1b-1024i (4 ms)
[ RUN      ] TestScannerThreading/TestScanner.ScanOrderingFail/1Serial2d16b1024r
/arrow/cpp/src/arrow/dataset/scanner_test.cc:1040: Failure
Expected equality of these values:
  next.status().message()
    Which is: "Data is not ordered\n/arrow/cpp/src/arrow/acero/assert_order_node.cc:190  AssertInBatchOrder(*record_batch, ordering_, comparators_)\n/arrow/cpp/src/arrow/acero/source_node.cc:158  output_->InputReceived(this, std::move(batch))"
  "Data is not ordered"
With diff:
@@ -1,3 @@
 Data is not ordered
-/arrow/cpp/src/arrow/acero/assert_order_node.cc:190  AssertInBatchOrder(*record_batch, ordering_, comparators_)
-/arrow/cpp/src/arrow/acero/source_node.cc:158  output_->InputReceived(this, std::move(batch))

There seems to be some stray output on that specific job.

@EnricoMi
EnricoMi force-pushed the assert-order-node branch 2 times, most recently from 7750094 to 2e30308 Compare November 20, 2024 10:54
@EnricoMi

Copy link
Copy Markdown
Collaborator Author

@raulcd thanks, assertions adjusted

@westonpace westonpace left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Conceptually this makes sense to me. I like that it verifies the ordering and it's good use of a sequencing queue. I can't speak much to the code in compare.h / compare.cc as I'm not very familiar with it.

It's a brand new node and I don't see much change to existing paths so I'm not too concerned about a regression.

Comment thread cpp/src/arrow/acero/exec_plan.h Resolved
Comment thread cpp/src/arrow/compare.cc Outdated
Comment on lines 879 to 911

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to the PR?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverted

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should you use !ordering.is_explicit()?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, fixed

@EnricoMi

Copy link
Copy Markdown
Collaborator Author

@westonpace saying: "I can't speak much to the code in compare.h / compare.cc as I'm not very familiar with it."

@pitrou @lidavidm @bkietz @zeroshade you seem to be familiar with compare.cc, can you take a look, please?

@EnricoMi
EnricoMi force-pushed the assert-order-node branch 2 times, most recently from e26b4af to 4dd5f29 Compare February 12, 2025 11:12
@EnricoMi
EnricoMi force-pushed the assert-order-node branch 2 times, most recently from a5737fd to 23823d9 Compare February 12, 2025 14:47
@EnricoMi EnricoMi changed the title GH-34698: [C++][Acero] Add node that emits explicit ordering after asserting order GH-34698: [C++][Python][Acero] Add node that emits explicit ordering after asserting order Feb 12, 2025
@EnricoMi
EnricoMi force-pushed the assert-order-node branch from 96684d2 to 3a51271 Compare March 10, 2025 06:17
@github-actions

Copy link
Copy Markdown

Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.

@github-actions github-actions Bot added the Status: stale-warning Issues and PRs flagged as stale which are due to be closed if no indication otherwise label Mar 10, 2026
@github-actions github-actions Bot closed this Mar 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: C++ Component: Python Status: stale-warning Issues and PRs flagged as stale which are due to be closed if no indication otherwise

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants