Skip to content

fix(ros2): stop connector executor without hanging on shutdown - #889

Open
maciejmajek wants to merge 3 commits into
mainfrom
fix/connector-shutdown-hang-v2
Open

maciejmajek wants to merge 3 commits into
mainfrom
fix/connector-shutdown-hang-v2

Conversation

@maciejmajek

@maciejmajek maciejmajek commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Purpose

  • ROS2BaseConnector.shutdown() sometimes blocks forever. Seen in CI on self-hosted and GitHub-hosted runners: the main thread waits in self._thread.join() and never returns.

Proposed Changes

  • Cause: the connector runs Executor.spin() in a thread and stops it with Executor.shutdown() from another thread. shutdown() destroys the executor's guard conditions and clears its nodes. If the spin thread is building its wait set at that moment, it waits on an empty set with no timeout, forever.
  • Fix: the thread runs executor.spin_until_future_complete(self._stop_spinning). shutdown() completes the future, which wakes the executor (rclpy adds future.add_done_callback(lambda x: self.wake())). The thread returns, then the executor is shut down with nobody spinning it.
  • shutdown() order: stop and join the thread, then TF listener, action and topic APIs, executor, and destroy the node last. Before, the node was destroyed while the executor could still use it.
  • Bump rai_core to 2.13.1.

Behaviour for existing code

  • Public rclpy API, same code path: spin() and spin_until_future_complete() both run _spin_once_impl per iteration. The only extra work is a future.done() check on wake.
  • No polling. Idle connectors sleep in the wait set like before.
  • Jazzy: the executor stays marked as spinning for the whole thread life, same as spin().
  • Demos that call rclpy.spin(connector.node) (rai_semap, rai_perception detection publisher) move the node to the global executor, same as today. The connector thread then waits on an empty executor and still wakes on shutdown.

Issues

  • CI hangs in tests/tools/ros2/test_topic_tools.py during connector teardown.

Testing

  • New test_ros2_connector_shutdown_does_not_hang: 100 connector create and shutdown cycles, 60 s limit.
  • Shutdown stress, pure rclpy, MultiThreadedExecutor, node with timer and 40 subscriptions, 4 busy threads, 4 CPUs:
hangs median shutdown
jazzy old 13 / 2500 56 to 68 ms
jazzy new 0 / 4000 11 to 19 ms
humble old 14 / 1500 32 ms
humble new 0 / 3000 14 ms
  • Topic, service and action benchmark (MultiThreadedExecutor): old and new are equal within noise on both distros.
jazzy old new
service call median 520 us 456 to 524 us
action round trip median 4 ms 4 ms
humble old new
topic latency median 122 us 102 us
service call median 404 us 398 us
action round trip median 5 ms 5 ms

@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@2bc2f32). Learn more about missing BASE report.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #889   +/-   ##
=======================================
  Coverage        ?   76.42%           
=======================================
  Files           ?       83           
  Lines           ?     3674           
  Branches        ?        0           
=======================================
  Hits            ?     2808           
  Misses          ?      866           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

maciejmajek and others added 2 commits November 6, 2026 17:28
ROS2BaseConnector.shutdown() could block forever joining the spin
thread. Executor.shutdown() from another thread can leave spin()
waiting on an empty wait set. Spin with spin_once(timeout_sec=0.1)
and a stop flag, stop the thread first and destroy the node last.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@maciejmajek
maciejmajek force-pushed the fix/connector-shutdown-hang-v2 branch from c2e123c to c4e30ec Compare November 6, 2026 16:28
spin_until_future_complete() wakes the executor as soon as the future
completes, so shutdown is immediate and idle connectors do not poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant