Skip to content

GH-33580: [C++] Support emit info in Substrait extension-multi and AsOfJoin - #14799

Merged
westonpace merged 14 commits into
apache:masterfrom
rtpsw:ARROW-18417
Jan 27, 2023
Merged

westonpace merged 14 commits into
apache:masterfrom
rtpsw:ARROW-18417

Conversation

@rtpsw

@rtpsw rtpsw commented Dec 1, 2022 •

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Dec 1, 2022

Copy link
Copy Markdown

@rtpsw

rtpsw commented Dec 1, 2022 •

Copy link
Copy Markdown
Contributor Author

@westonpace, as compared to westonpace#27, this PR is against master. Note that the current commit includes some unrelated changes to AsofJoinNode (notably backpressure support and an early version of deadlock avoidance) which you may or may not want to include. You may also want some additional tests.

@rtpsw

rtpsw commented Dec 21, 2022

Copy link
Copy Markdown
Contributor Author

Note about this recent commit:

In gdb, the error had this backtrace:

Thread 3 "arrow-substrait" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff07fe640 (LWP 6962)]
0x00007ffff29f2cdd in __dynamic_cast () from /lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007ffff29f2cdd in __dynamic_cast () from /lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff48d732b in arrow::internal::checked_cast<arrow::ConcreteFutureImpl*, arrow::FutureImpl*&> (value=@0x7ffff07fcdc8: 0x555555a4bae0)
    at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/util/checked_cast.h:38
#2  0x00007ffff48d386b in arrow::(anonymous namespace)::GetConcreteFuture (future=0x555555a4bae0) at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/util/future.cc:172
#3  0x00007ffff48d3ab8 in arrow::FutureImpl::MarkFinished (this=0x555555a4bae0) at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/util/future.cc:193
#4  0x0000555555785dfc in arrow::Future<arrow::internal::Empty>::DoMarkFinished (this=0x555555a4a5a8, res=...) at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/util/future.h:658
#5  0x0000555555784902 in arrow::Future<arrow::internal::Empty>::MarkFinished<arrow::internal::Empty, void> (this=0x555555a4a5a8, s=...)
    at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/util/future.h:409
#6  0x00007ffff4a93f6b in arrow::compute::AsofJoinNode::CheckEnded()::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (__closure=0x7ffff07fcf20)
    at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:972
#7  0x00007ffff4a9c1a4 in arrow::compute::AsofJoinNode::Defer<arrow::compute::AsofJoinNode::CheckEnded()::{lambda()#1}::operator()() const::{lambda()#1}>::~Defer() (this=0x7ffff07fcf20, 
    __in_chrg=<optimized out>) at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:966
#8  0x00007ffff4a93ee3 in arrow::compute::AsofJoinNode::CheckEnded()::{lambda()#1}::operator()() const (__closure=0x7ffff07fd0b0)
    at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:975
#9  0x00007ffff4ab01ef in std::__invoke_impl<arrow::Status, arrow::compute::AsofJoinNode::CheckEnded()::{lambda()#1}&>(std::__invoke_other, arrow::compute::AsofJoinNode::CheckEnded()::{lambda()#1}&) (
    __f=...) at /usr/include/c++/11/bits/invoke.h:61
#10 0x00007ffff4aac13e in std::__invoke_r<arrow::Status, arrow::compute::AsofJoinNode::CheckEnded()::{lambda()#1}&>(arrow::compute::AsofJoinNode::CheckEnded()::{lambda()#1}&) (__fn=...)
    at /usr/include/c++/11/bits/invoke.h:116
#11 0x00007ffff4aa65e5 in std::_Function_handler<arrow::Status (), arrow::compute::AsofJoinNode::CheckEnded()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/11/bits/std_function.h:291
#12 0x00007ffff4ad109e in std::function<arrow::Status ()>::operator()() const (this=0x7ffff07fd0b0) at /usr/include/c++/11/bits/std_function.h:590
#13 0x00007ffff4abf832 in arrow::compute::(anonymous namespace)::ExecPlanImpl::ScheduleTask(std::function<arrow::Status()>) (this=0x555555a495e0, fn=...)
    at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:94
#14 0x00007ffff4ac17fe in arrow::compute::ExecPlan::ScheduleTask(std::function<arrow::Status ()>) (this=0x555555a495e0, fn=...)
    at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:385
#15 0x00007ffff4a94059 in arrow::compute::AsofJoinNode::CheckEnded (this=0x555555a4a500) at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:971
#16 0x00007ffff4a94235 in arrow::compute::AsofJoinNode::Process (this=0x555555a4a500) at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:983
#17 0x00007ffff4a94597 in arrow::compute::AsofJoinNode::ProcessThread (this=0x555555a4a500) at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:1025
#18 0x00007ffff4a945c2 in arrow::compute::AsofJoinNode::ProcessThreadWrapper (node=0x555555a4a500) at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:1031
#19 0x00007ffff4ab8f48 in std::__invoke_impl<void, void (*)(arrow::compute::AsofJoinNode*), arrow::compute::AsofJoinNode*> (
    __f=@0x555555a4a960: 0x7ffff4a945a6 <arrow::compute::AsofJoinNode::ProcessThreadWrapper(arrow::compute::AsofJoinNode*)>) at /usr/include/c++/11/bits/invoke.h:61
#20 0x00007ffff4ab8ecd in std::__invoke<void (*)(arrow::compute::AsofJoinNode*), arrow::compute::AsofJoinNode*> (
    __fn=@0x555555a4a960: 0x7ffff4a945a6 <arrow::compute::AsofJoinNode::ProcessThreadWrapper(arrow::compute::AsofJoinNode*)>) at /usr/include/c++/11/bits/invoke.h:96
#21 0x00007ffff4ab8e2d in std::thread::_Invoker<std::tuple<void (*)(arrow::compute::AsofJoinNode*), arrow::compute::AsofJoinNode*> >::_M_invoke<0ul, 1ul> (this=0x555555a4a958)
    at /usr/include/c++/11/bits/std_thread.h:253
#22 0x00007ffff4ab8de2 in std::thread::_Invoker<std::tuple<void (*)(arrow::compute::AsofJoinNode*), arrow::compute::AsofJoinNode*> >::operator() (this=0x555555a4a958)
    at /usr/include/c++/11/bits/std_thread.h:260
#23 0x00007ffff4ab8dc2 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(arrow::compute::AsofJoinNode*), arrow::compute::AsofJoinNode*> > >::_M_run (this=0x555555a4a950)
    at /usr/include/c++/11/bits/std_thread.h:211
#24 0x00007ffff2a222b3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#25 0x00007ffff27b2b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#26 0x00007ffff2844a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

I observed that gdb is able to statically cast to ConcreteFutureImpl* but fails to dynamically cast to it:

(gdb) frame 6
#6  0x00007ffff4a93f6b in arrow::compute::AsofJoinNode::CheckEnded()::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const (__closure=0x7ffff07fcf20)
    at /mnt/user1/tscontract/github/rtpsw/arrow/cpp/src/arrow/compute/exec/asof_join_node.cc:972
972             Defer cleanup([this]() { finished_.MarkFinished(); });
(gdb) print dynamic_cast<arrow::ConcreteFutureImpl*>(finished_.impl_.get())
Couldn't determine value's most derived type for dynamic_cast

This suggests something with the RTTI of ConcreteFutureImpl is not right. I made several attempts to figure out the root cause but couldn't as yet. I therefore propose in this commit to use a static_cast to ConcreteFutureImpl* as a workaround.

@rtpsw

rtpsw commented Dec 21, 2022

Copy link
Copy Markdown
Contributor Author

I looked into the ASAN failure, and I suspect that it is because ScheduleTask is synchronous. In particular, I observed that the ScheduleTask invocation in CheckEnded is executed within the process-thread of the AsofJoinNode, which marks finished and leads to an invocation of ~AofJoinNode within the same thread. This is why a detach invocation was added in this destructor, but this desctructor shouldn't be called from the process-thread. The actual ASAN failure happened in a later invocation of CheckEnded from the process-thread, which doesn't expect this destructor to have been called. @westonpace, I suspect this means that, at least with the current threading structure of AsofJoinNode, we would need to ensure ScheduleTask is not synchronous - WDYT?

@rtpsw

rtpsw commented Dec 22, 2022 •

Copy link
Copy Markdown
Contributor Author

The recent commit fixed the ASAN failure by avoiding a null executor, so that the AsofJoinNode destructor is never called from the process-thread. It also includes a fix in exec_plan.cc that avoids hanging tasks when a later one fails to initialize. I observed such a condition when the source nodes hanged when the as-of-join node failed (as expected in some test case) to initialize with a null executor.

Note that accepting this commit would cause any code out there that uses as-of-join with a null executor (as in the current default exec context) to fail. My understanding is that the API is experimental right now, so it should be OK to break for a good reason.

@rtpsw

rtpsw commented Dec 22, 2022

Copy link
Copy Markdown
Contributor Author

The CI failures appear to be unrelated.

@rtpsw

rtpsw commented Jan 15, 2023

Copy link
Copy Markdown
Contributor Author

The CI job failure appears to be unrelated,

@rtpsw
rtpsw requested a review from westonpace January 15, 2023 09:36
@westonpace

Copy link
Copy Markdown
Member

@rtpsw this will need a rebase. Some of the changes here were brought in with the backpressure change. Would you like me to do this or do you want to?

@westonpace westonpace changed the title ARROW-18417: [C++] Support emit info in Substrait extension-multi and AsOfJoin GH-33580: [C++] Support emit info in Substrait extension-multi and AsOfJoin Jan 20, 2023
@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #33580 has been automatically assigned in GitHub to PR creator.

@rtpsw

rtpsw commented Jan 21, 2023

Copy link
Copy Markdown
Contributor Author

@rtpsw this will need a rebase. Some of the changes here were brought in with the backpressure change. Would you like me to do this or do you want to?

I just rebased. I didn't try to take out the backpressure changes, but let me know if you want that. I left the backpressure_counter_ initial value as 1, though I recall we discussed 0 would work too.

@westonpace westonpace left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can clean this up a little bit but otherwise it looks pretty solid.

@rtpsw

rtpsw commented Jan 26, 2023

Copy link
Copy Markdown
Contributor Author

@westonpace, I rebased again after fixing per your requests.

@rtpsw
rtpsw requested a review from westonpace January 26, 2023 08:37

@westonpace westonpace left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I think we're good now.

@westonpace
westonpace merged commit 108a028 into apache:master Jan 27, 2023
@ursabot

ursabot commented Jan 27, 2023

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = fc55807 and contender = 108a028. 108a028 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.51% ⬆️0.0%] test-mac-arm
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 108a0286 ec2-t3-xlarge-us-east-2
[Failed] 108a0286 test-mac-arm
[Failed] 108a0286 ursa-i9-9960x
[Finished] 108a0286 ursa-thinkcentre-m75q
[Finished] fc558071 ec2-t3-xlarge-us-east-2
[Failed] fc558071 test-mac-arm
[Failed] fc558071 ursa-i9-9960x
[Finished] fc558071 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Support emit info in Substrait extension-multi and AsOfJoin

3 participants