Skip to content

address context shutdown racy condition. - #3219

Merged
skyegalaxy merged 1 commit into
lyricalfrom
fujitatomoya/lyrical/racy-context-shutdown
Aug 12, 2026
Merged

skyegalaxy merged 1 commit into
lyricalfrom
fujitatomoya/lyrical/racy-context-shutdown

Conversation

@fujitatomoya

Copy link
Copy Markdown
Collaborator

Description

retarget #3217 to lyrical

Fixes # (issue)

Is this user-facing behavior change?

Did you use Generative AI?

Additional Information

Signed-off-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
@fujitatomoya fujitatomoya self-assigned this Aug 11, 2026

@fujitatomoya fujitatomoya left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is ABI compatibility change, so that we can put this to lyrical and downstream branches.

@mergify

mergify Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@fujitatomoya

Copy link
Copy Markdown
Collaborator Author

Pulls: #3219
Gist: https://gist.githubusercontent.com/fujitatomoya/977ef70875dc4467b6f510f9db8e39a5/raw/2cb0f02adf8bf5ad334675d3d51445cf204e17ed/ros2.repos
BUILD args: --packages-above-and-dependencies rclcpp
TEST args: --packages-above rclcpp
ROS Distro: lyrical
Job: ci_launcher
ci_launcher ran: https://ci.ros2.org/job/ci_launcher/20058

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Linux-rhel Build Status
  • Windows Build Status

@skyegalaxy skyegalaxy left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm with green CI

@fujitatomoya

Copy link
Copy Markdown
Collaborator Author
  • Linux Build Status
  • Linux-aarch64 Build Status
  • Linux-rhel Build Status
  • Windows Build Status

@skyegalaxy
skyegalaxy merged commit b723878 into lyrical Aug 12, 2026
3 checks passed
@skyegalaxy

Copy link
Copy Markdown
Member

https://github.com/Mergifyio backport humble jazzy kilted

@mergify

mergify Bot commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

backport humble jazzy kilted

✅ Backports have been created

Details

Cherry-pick of b723878 has failed:

On branch mergify/bp/humble/pr-3219
Your branch is up to date with 'origin/humble'.

You are currently cherry-picking commit b723878.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   rclcpp/src/rclcpp/signal_handler.cpp

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   rclcpp/src/rclcpp/context.cpp
	deleted by us:   rclcpp/test/rclcpp/test_context.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Cherry-pick of b723878 has failed:

On branch mergify/bp/jazzy/pr-3219
Your branch is up to date with 'origin/jazzy'.

You are currently cherry-picking commit b723878.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   rclcpp/src/rclcpp/signal_handler.cpp
	modified:   rclcpp/test/rclcpp/test_context.cpp

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   rclcpp/src/rclcpp/context.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Cherry-pick of b723878 has failed:

On branch mergify/bp/kilted/pr-3219
Your branch is up to date with 'origin/kilted'.

You are currently cherry-picking commit b723878.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   rclcpp/src/rclcpp/signal_handler.cpp
	modified:   rclcpp/test/rclcpp/test_context.cpp

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   rclcpp/src/rclcpp/context.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

fujitatomoya added a commit that referenced this pull request Aug 13, 2026
Signed-off-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
(cherry picked from commit b723878)

# Conflicts:
#	rclcpp/src/rclcpp/context.cpp
fujitatomoya added a commit that referenced this pull request Aug 13, 2026
Signed-off-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
(cherry picked from commit b723878)

# Conflicts:
#	rclcpp/src/rclcpp/context.cpp
fujitatomoya added a commit that referenced this pull request Aug 13, 2026
Signed-off-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
(cherry picked from commit b723878)

# Conflicts:
#	rclcpp/src/rclcpp/context.cpp
#	rclcpp/test/rclcpp/test_context.cpp
fujitatomoya added a commit that referenced this pull request Aug 13, 2026
Signed-off-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
(cherry picked from commit b723878)

# Conflicts:
#	rclcpp/src/rclcpp/context.cpp
#	rclcpp/test/rclcpp/test_context.cpp
ahcorde pushed a commit that referenced this pull request Aug 21, 2026
Signed-off-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
(cherry picked from commit b723878)

Co-authored-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
ahcorde pushed a commit that referenced this pull request Aug 21, 2026
Signed-off-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
(cherry picked from commit b723878)

Co-authored-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
jmachowinski pushed a commit that referenced this pull request Aug 22, 2026
Signed-off-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
(cherry picked from commit b723878)

# Conflicts:
#	rclcpp/src/rclcpp/context.cpp

Co-authored-by: Tomoya Fujita <fujita.tomoya@triorb.co.jp>
@ahcorde
ahcorde deleted the fujitatomoya/lyrical/racy-context-shutdown branch September 16, 2026 07:45
jon-mcmillan pushed a commit to polymathrobotics/livekit_ros2_bridge that referenced this pull request Sep 23, 2026
`test_subscription_lease_manager` sometimes crashes in CI right after
all its tests pass. Our test fixtures stop ROS during program exit, and
a recent rclcpp update (ros2/rclcpp#3219) made that unsafe: by then,
memory rclcpp needs for shutdown has already been freed. This change
stops ROS at the end of each test suite instead, while the program is
still running.

## What changed

- A new fixture base in the test helpers initializes rclcpp when a test
suite starts and shuts it down when the suite ends, so shutdown always
happens while `main()` is still running.
- Every fixture that used the static init now inherits from that base,
not just the lease manager ones. The others don't crash today, but they
could as soon as a test adds an earlier ROS shutdown.
- Per-test `ScopedRclcppInit` locals are left alone, since they're
destroyed inside the test body.

I reproduced the crash locally against rclcpp 16.0.21, where it hit
within a few runs. With this change, the full suite passes and 600
stress runs of the lease manager test finished without a crash.
gasmith added a commit to foxglove/foxglove-sdk that referenced this pull request Oct 13, 2026
### Changelog

None

### Docs

None

### Description

`smoke_test` intermittently exits with SIGSEGV (-11) on Jazzy and Kilted
after every test passes and the bridge logs "Shutdown complete". It
happens on `main` too.

- **Cause:** recent rclcpp releases
([ros2/rclcpp#3219](ros2/rclcpp#3219),
backported to Jazzy 28.1.22 and Kilted 29.5.11) track in-progress
context shutdowns in a `thread_local` set. The test's `main()` returned
without calling `rclcpp::shutdown()`, so the global context was shut
down by its static destructor during `exit()`, after glibc had already
destroyed the main thread's `thread_local` objects.
`spinReturnsOnContextShutdown` had created that set by shutting down a
context on the main thread, so the exit-time shutdown inserts into freed
memory. Whether that crashes depends on what the allocator left there.
- **Fix:** destroy the bridge and executor while ROS is still valid,
then call `rclcpp::shutdown()` before returning. This matches the
production node's `main()`, which isn't affected.
- **Verification:** reproduced locally with core dumps showing this
stack, plus a small rclcpp-only program. With freed memory forced to
garbage, the crash happens nearly every run on Jazzy and Kilted before
the fix and never after. An ASan+UBSan build reports nothing through
exit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants