Repository navigation
Conversation
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or In the case of PARQUET issues on JIRA the title also supports: See also: |
e66878f to
937695d
Compare
|
|
raulcd
left a comment
There was a problem hiding this comment.
I am probably not qualified to review this thoroughly but the CI failure is related:
[ FAILED ] TestScannerThreading/TestScanner.ScanOrderingFail/0Serial1d1b1024r, where GetParam() = serial-1d-1b-1024i (4 ms)
[ RUN ] TestScannerThreading/TestScanner.ScanOrderingFail/1Serial2d16b1024r
/arrow/cpp/src/arrow/dataset/scanner_test.cc:1040: Failure
Expected equality of these values:
next.status().message()
Which is: "Data is not ordered\n/arrow/cpp/src/arrow/acero/assert_order_node.cc:190 AssertInBatchOrder(*record_batch, ordering_, comparators_)\n/arrow/cpp/src/arrow/acero/source_node.cc:158 output_->InputReceived(this, std::move(batch))"
"Data is not ordered"
With diff:
@@ -1,3 @@
Data is not ordered
-/arrow/cpp/src/arrow/acero/assert_order_node.cc:190 AssertInBatchOrder(*record_batch, ordering_, comparators_)
-/arrow/cpp/src/arrow/acero/source_node.cc:158 output_->InputReceived(this, std::move(batch))
There seems to be some stray output on that specific job.
7750094 to
2e30308
Compare
|
@raulcd thanks, assertions adjusted |
2e30308 to
af38189
Compare
westonpace
left a comment
There was a problem hiding this comment.
Thanks! Conceptually this makes sense to me. I like that it verifies the ordering and it's good use of a sequencing queue. I can't speak much to the code in compare.h / compare.cc as I'm not very familiar with it.
It's a brand new node and I don't see much change to existing paths so I'm not too concerned about a regression.
There was a problem hiding this comment.
Is this change related to the PR?
There was a problem hiding this comment.
Should you use !ordering.is_explicit()?
@pitrou @lidavidm @bkietz @zeroshade you seem to be familiar with compare.cc, can you take a look, please? |
9aed89f to
59b4afa
Compare
e26b4af to
4dd5f29
Compare
a5737fd to
23823d9
Compare
…De]sc assertions robust
Co-authored-by: Weston Pace <weston.pace@gmail.com>
96684d2 to
3a51271
Compare
|
Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer. |
Rationale for this change
An acero node that turns an implicit ordering into an explicit ordering (rows sorted by some columns) is useful to re-use order that already exists in the data.
What changes are included in this PR?
This PR adds the
AssertOrderNodethat implements this logic. TheScanneremploys that node to turn the implicit ordering of theScanNodeinto an explicit order as defined by user code viaScanBuilder.Ordering.Are these changes tested?
There are unit tests for the
AssertOrderNodeas well as for theScanNodeandScanBuilder.Are there any user-facing changes?
The following options has been added to C++:
orderingoption added toScanOptionsOrderingoption added toScannerBuilderAssertNodeOptionsclassThe following method is added to Python
Dataset:assert_sortedasserts given order