You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Only one timer + MultiThreadedExecutor will block executor #877
On Dashing, will block executor, and cannot be recover
On Master, will block until next_exec_timeout_
MultiThreadedExecutor::run(size_t)
{
while (rclcpp::ok(this->context_) && spinning.load()) {
executor::AnyExecutable any_exec;
{
std::lock_guard<std::mutex> wait_lock(wait_mutex_);
if (!rclcpp::ok(this->context_) || !spinning.load()) {
return;
}
//BUG TODO: this will own wait_mutex_, will block scheduled_timers_ remove
if (!get_next_executable(any_exec, next_exec_timeout_)) {
continue;
}
if (any_exec.timer) {
// Guard against multiple threads getting the same timer.
if (scheduled_timers_.count(any_exec.timer) != 0) {
// Make sure that any_exec's callback group is reset before
// the lock is released.
if (any_exec.callback_group) {
any_exec.callback_group->can_be_taken_from().store(true);
}
continue;
}
scheduled_timers_.insert(any_exec.timer);
}
}
if (yield_before_execute_) {
std::this_thread::yield();
}
execute_any_executable(any_exec);
if (any_exec.timer) {
//BUG TODO: scheduled_timers_.erase need wait_mutex_ release.
std::lock_guard<std::mutex> wait_lock(wait_mutex_);
auto it = scheduled_timers_.find(any_exec.timer);
if (it != scheduled_timers_.end()) {
scheduled_timers_.erase(it);
}
}
// Clear the callback_group to prevent the AnyExecutable destructor from
// resetting the callback group `can_be_taken_from`
any_exec.callback_group.reset();
}
}
This interrupt in Executor::execute_any_executable:
if (rcl_trigger_guard_condition(&interrupt_guard_condition_) != RCL_RET_OK) {
throw std::runtime_error(rcl_get_error_string().str);
}
exist to avoid a thread, in the case the executor is multithreaded and the threads are not mutually exclusive, waiting indefinitely in rcl_wait in wait.c in rcl. When thread 1 goes into wait_for_work and adds the timer handle to wait_set and is woken by the timer, it starts executing the timer. At the same time, the lock is released and the other thread waiting, thread 2, goes into 'wait_for_work'. Since thread 1 is executing the timer callback, the memory strategy does not add the timer handle to wait_set and thread 2 goes into rcl_wait with a wait_set that has no timer handle, and when thread 1 finishes executing the timer callback it is waiting for thread 2 to unlock;
effectively, having thread 1 and 2 waiting indefinitely. With the interrupt_guard_condition triggered, any thread waiting in rcl_wait will wake up and look for work to be executed. You can run the code given to see this in effect:
If the interrupt guard condition did not exist, your solution will still not work because the thread will be waiting in rcl_wait indefinitely, and even if the thread can erase the timer from the scheduled_timers_ set while the other thread is asleep, the thread in rcl_wait will not wake up. scheduled_timers_ is solely used to keep track of the timers used. I tested the multithreaded executor with one timer to verify. It does what is expected.
@peterpena I don't see the connection between your comment and @liqinghua bug report. @liqinghua I'm closing this due to long time inactivity, we can reopen the issue if you provide an example of the problem.
PS: There was a thread safety problem that made MultiThreadedExecutor hung when using timers. That was fixed and backported to Dashing #869 at about the same time that @liqinghua opened this issue. This is likely a duplicate of that (already fixed) error.
Activity
liqinghua commentedon Sep 27, 2019
On Dashing version, fixing suggestion
ivanpauno commentedon Sep 27, 2019
Could you provide an example of the problem?
peterpena commentedon May 29, 2020
This interrupt in
Executor::execute_any_executable:exist to avoid a thread, in the case the executor is multithreaded and the threads are not mutually exclusive, waiting indefinitely in
rcl_waitinwait.cinrcl. When thread 1 goes intowait_for_workand adds the timer handle towait_setand is woken by the timer, it starts executing the timer. At the same time, the lock is released and the other thread waiting, thread 2, goes into 'wait_for_work'. Since thread 1 is executing the timer callback, the memory strategy does not add the timer handle towait_setand thread 2 goes intorcl_waitwith await_setthat has no timer handle, and when thread 1 finishes executing the timer callback it is waiting for thread 2 to unlock;effectively, having thread 1 and 2 waiting indefinitely. With the
interrupt_guard_conditiontriggered, any thread waiting inrcl_waitwill wake up and look for work to be executed. You can run the code given to see this in effect:If the interrupt guard condition did not exist, your solution will still not work because the thread will be waiting in
rcl_waitindefinitely, and even if the thread can erase the timer from thescheduled_timers_set while the other thread is asleep, the thread inrcl_waitwill not wake up.scheduled_timers_is solely used to keep track of the timers used. I tested the multithreaded executor with one timer to verify. It does what is expected.ivanpauno commentedon May 29, 2020
@peterpena I don't see the connection between your comment and @liqinghua bug report.
@liqinghua I'm closing this due to long time inactivity, we can reopen the issue if you provide an example of the problem.
PS: There was a thread safety problem that made
MultiThreadedExecutorhung when using timers. That was fixed and backported to Dashing #869 at about the same time that @liqinghua opened this issue. This is likely a duplicate of that (already fixed) error.Move the guard condition cleanup after removing callback. (ros2#877)
Fix TSA warnings when building with clang thread analysis. (ros2#877)