Repository navigation
feat(rai_core): numpy-2-safe image bridge; drop ROS 2 Humble and Python 3.10 - #895
Conversation
… cv_bridge ROS 2 Jazzy's apt cv_bridge cannot run under numpy 2: its compiled helper (cv_bridge_boost) is built against Ubuntu 24.04's numpy 1.26, `import cv_bridge` swallows the import failure, and the first imgmsg_to_cv2 call then segfaults the interpreter. Big-endian images also hit ndarray.newbyteorder(), which numpy 2 removed. This is why rai_core had to cap numpy below 2, which in turn held opencv-python at 4.11. rai.communication.ros2.image_bridge.CvBridge is a pure numpy + OpenCV drop-in for the cv_bridge methods RAI uses (imgmsg_to_cv2, cv2_to_imgmsg, compressed_imgmsg_to_cv2, cv2_to_compressed_imgmsg). It follows cv_bridge's encodings, colour conversions, 8<->16 bit scaling and refusals, and OpenCV's saturate_cast for float->integer depth changes (NaN, +-inf and values beyond int32 become INT_MIN before saturation, so invalid depth pixels become 0, not the maximum). A parity harness against the real cv_bridge (1595 cases: every source x target encoding, padding, big-endian, compressed, errors; plus 22 special-value cases) matches 100% under numpy 1.26/OpenCV 4.11 and numpy 2.5/OpenCV 4.14. One deliberate difference: rows are read with `step` as the exact row stride. cv_bridge assumes padding is a whole number of pixels and returns a scrambled image otherwise; a step shorter than a row now raises instead of silently cropping. All cv_bridge imports in rai_core, rai_perception, rai_semap and the tests now use the new module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ROS 2 Humble (Python 3.10) is no longer supported; ROS 2 Jazzy on Python 3.12 is. - requires-python ">=3.12,<3.13" for the workspace and rai_core, rai_perception, rai_semap, rai_s2s, rai_sim, rai_bench, rai_whoami. rai_finetune is not part of the uv workspace and is unchanged. - rai_core: numpy ">=1.26.0,<3.0.0" (was "<2.0"), now that images no longer go through cv_bridge; opencv-python(-headless) "<6" (was "<5"). The Python-3.10-only pillow and streamlit variants are gone; rai_s2s likewise drops its 3.10 onnxruntime variants. - rai_perception and rai_semap require rai_core>=2.15.0 (image_bridge). - Versions (minor: Python 3.10/Humble users stay on the previous releases through requires-python): rai_core 2.15.0, rai_perception 0.5.0, rai_semap 0.2.0, rai_s2s 1.2.0, rai_bench 0.5.0, rai_sim 0.1.0, rai_whoami 0.3.0. - uv.lock re-locked; only the Python 3.10 forks are removed. The workspace lock itself stays on numpy 1.26/OpenCV 4.11 because the nomad group's visualnav-transformer requires numpy<2. - ROS2ServiceAPI always passes timeout_sec (the Humble-only branch is removed); Humble removed from CI (test matrix, pre-commit container), download_demo.sh, test notes and the docs. The pre-commit job now runs in the Jazzy container, whose Python is externally managed (PEP 668), so it installs pre-commit into a venv. The test suite passes on Jazzy both as locked (numpy 1.26, OpenCV 4.11) and with numpy 2.5 + OpenCV 5.0 installed: 930 passed, 0 failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #895 +/- ##
==========================================
+ Coverage 76.40% 77.28% +0.88%
==========================================
Files 83 84 +1
Lines 3670 3848 +178
==========================================
+ Hits 2804 2974 +170
- Misses 866 874 +8 ☔ View full report in Codecov by Harness. |
maciejmajek
left a comment
There was a problem hiding this comment.
Nice work, the core of this holds up well. I checked a few things independently.
Bridge parity. I compared image_bridge.CvBridge with the real cv_bridge from ROS 2 Lyrical (numpy 2.3.5). These all came out identical:
imgmsg_to_cv2on 812 source x target encoding pairs (passthrough, genericnUCm, bare depths, Bayer, YUV, NaN/inf, signed/unsigned 8/16 bit)cv2_to_imgmsgon sliced, Fortran-ordered, big-endian and odd-shaped arrays- compression in jpg/jpeg/png/tiff/bmp/webp, and decoding of grayscale, 16-bit and alpha PNGs
- which methods return a view of
msg.dataand which return a copy
Big-endian input works in the new bridge, while Lyrical's cv_bridge raises AttributeError on it under numpy 2.
numpy 2 with the rest of Jazzy. In osrf/ros:jazzy-desktop-full, I scanned all 464 Python extension modules. cv_bridge is the only ROS package that uses the numpy C API. Message bindings, rclpy and rosbag2_py include numpy headers but never call import_array. Under numpy 2.0.2 and 2.3.5, all 156 ROS Python packages import the same way as under 1.26. These all work: fixed-array message serialization (CameraInfo, Imu, Odometry), a real rclpy pub/sub round trip, point_cloud2, tf2_ros + tf2_geometry_msgs, image_geometry, laser_geometry, rosbag2_py MCAP write/read, message_filters and nav2_simple_commander. Only the apt cv_bridge.imgmsg_to_cv2 segfaults, as the PR says.
Test suite. I re-ran the suite in osrf/ros:jazzy-desktop-full following the CI steps (vcs import, rosdep, colcon build, CPU torch, jackd). As locked (numpy 1.26.4, OpenCV 4.11): 955 passed, 4 skipped, 11 xfailed, 0 failed. With numpy 2.5.4 and OpenCV 5.0.0 installed over the locked set: 955 passed, 4 skipped, 11 xfailed, 0 failed.
Semver. Minor bumps look right to me under CONTRIBUTING.md. Nothing in the public API, ROS interfaces or config schemas breaks, and requires-python keeps Python 3.10 / Humble users on the previous releases. Removing rai_nomad (see inline) doesn't change that: it is a colcon extension, not part of the published rai_core API.
CONTRIBUTING.md asks for breaking changes to be documented in the release notes, so please add these when releasing rai_core 2.15.0:
- Dropped: Python 3.10, ROS 2 Humble, Ubuntu 22.04, and the NoMaD integration (rai_nomad,
--group nomad). - Any code that still imports
cv_bridge(your own nodes, other libraries in the same process) will segfault on the firstimgmsg_to_cv2once numpy 2 is installed, and theimportgives no warning. Migrate tofrom rai.communication.ros2.image_bridge import CvBridge. That class covers the four conversion methods. It does not havecv_bridge's helperscvtype2_to_dtype_with_channels,encoding_to_cvtype2,numpy_type_to_cvtype,cvdepth_to_numpy_depthorcvtype_to_name. - With the
ros2 run/ros2 launchsetup from the docs (system Python with the venv onPYTHONPATH), venv numpy 2 shadows the apt numpy. Apt-only modules then fail to import: scipy ("numpy.dtype size changed") and matplotlib ("numpy.core.multiarray failed to import"). Install them into the venv. - Error behaviour changes (details inline):
- Conversion errors are now
rai.communication.ros2.image_bridge.CvBridgeError. It still subclassesTypeError, butexcept cv_bridge.CvBridgeErrorno longer catches it. - A
stepshorter than one row now raises instead of silently cropping. - An undecodable compressed image with a non-passthrough encoding now raises instead of returning
None.
- Conversion errors are now
Small nit outside the diff: src/rai_finetune/README.md:59 and src/rai_finetune/pyproject.toml:8 still say Python 3.10 was chosen "for compatibility with the rest of RAI components". After this PR, rai_finetune (>=3.10,<3.12) and the rest (==3.12) share no Python version, so that sentence is now wrong. Either update the wording or move rai_finetune to 3.12.
Blocking for me: dropping nomad together with the lock move, CI coverage for the numpy version the lock doesn't use, and the rai_perception pin in rai_semap (all inline).
The nomad dependency group (visualnav-transformer, which requires numpy<2) was the only thing keeping the workspace lock on numpy 1.26. The integration is low value and the model is outdated, so it goes: the nomad group, the diffusion-policy override, rai_nomad, its docs page, and every reference in the docs, READMEs, NOTICE, the configurator and the rai_semap design notes. uv.lock is re-locked with `uv lock -P numpy -P opencv-python -P opencv-python-headless`: 84 nomad-only packages are removed, numpy moves 1.26.4 -> 2.5.4 and opencv-python(-headless) 4.11.0.86 -> 5.0.0.93, and nothing else changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rai_perception 0.4.0 still imports the apt cv_bridge, which segfaults under numpy 2. Pin the release that uses rai's image bridge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uv.lock is now on numpy 2, but rai_core still allows numpy 1.26, which is what ROS 2 Jazzy ships through apt. Add a matrix leg that installs numpy 1.26.4 with opencv-python(-headless) <4.12 (4.12+ needs numpy 2) on top of the locked set, so both ends of the range are tested. Coverage is uploaded from the locked leg only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rai_finetune said Python 3.10 was chosen for compatibility with the rest of RAI, which now requires Python 3.12. Say that it needs its own virtual environment instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…matrix None of the pre-commit hooks need ROS, so the job no longer runs in the jazzy-desktop-full container: no image pull, no apt step inside the container, and no PEP 668 venv workaround. Python comes from setup-python, and shellcheck (already on the runner image) is installed only if missing. The `vcs import` step is dropped: pre-commit --all-files only checks tracked files, so the imported repos were never linted. build-and-test sets fail-fast: false so a failure in one numpy leg does not cancel the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kt3k/license_checker builds an ubuntu:latest image and apt-installs curl, git and unzip on every run, so the check fails whenever the runner cannot reach the Ubuntu mirrors (it did on both runs of this PR). Run the same checker (license_checker v3.1.2, same .licenserc.json) in denoland/deno:alpine-1.46.3, pinned by digest. Locally it checks the same 423 files and still fails on a file without the header. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Support only numpy 2 instead of both 1.26 and 2, so there is one configuration to test and maintain. rai_core now requires numpy>=2.0.0,<3 and opencv-python(-headless)>=4.10.0.84: 4.10.0.84 is the first opencv-python build that imports under numpy 2 (4.9.0.80 and 4.10.0.82 fail with "numpy.core.multiarray failed to import"). The numpy 1.26 CI leg is no longer needed, so build-and-test goes back to a single job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move rai_core to numpy 2, drop ROS 2 Humble / Python 3.10, and drop the NoMaD integration.
Why
ROS 2 Jazzy's apt
cv_bridgeis compiled against Ubuntu 24.04's numpy 1.26. Under numpy 2,import cv_bridgehides the import failure, andimgmsg_to_cv2then segfaults the process. That is why rai_core capped numpy below 2, and that cap is what kept OpenCV at 4.11 for every downstream user.Changes
rai.communication.ros2.image_bridge.CvBridgereplacescv_bridgefor the methods RAI uses. It is pure numpy + OpenCV and works under numpy 1 and 2. Every import in rai_core, rai_perception, rai_semap and the tests now uses it.stepshorter than a row raises an error instead of cropping silently.>=2.0,<3and opencv-python(-headless)>=4.10.0.84,<6(4.10.0.84 is the first OpenCV build that imports under numpy 2). Only numpy 2 is supported, so there is one configuration to test. requires-python is>=3.12,<3.13for the workspace and its packages; rai_finetune is unchanged because it is not a workspace member.download_demo.sh, a service-call branch, the test notes and the docs.nomaddependency group, the diffusion-policy override,rai_nomad, and its docs. Its visualnav-transformer required numpy<2 and was the only thing holding the lock on numpy 1.26.uv.lockis on numpy 2.5.4 / OpenCV 5.0.0 (re-locked withuv lock -P numpy -P opencv-python -P opencv-python-headless). 84 nomad-only packages are removed and nothing else changes.Release notes
rai_nomad,--group nomad). Projects that need numpy<2 stay on rai_core 2.14.cv_bridgein the same process segfaults on the firstimgmsg_to_cv2once numpy 2 is installed. Migrate tofrom rai.communication.ros2.image_bridge import CvBridge.ros2 run/ros2 launch(system Python with the venv onPYTHONPATH), venv numpy 2 shadows the apt numpy, so apt-only scipy and matplotlib fail to import. Install them into the venv.rai.communication.ros2.image_bridge.CvBridgeError(aTypeError), notcv_bridge.CvBridgeError. Some failures thatcv_bridgereported asKeyErrororcv2.errornow raise it too.Tested
cv_bridgeuses numpy's C API. Message serialization, rclpy pub/sub, tf2, rosbag2, point_cloud2, image_geometry and laser_geometry all work under numpy 2.0 and 2.3.🤖 Generated with Claude Code