Repository navigation
feat(pixi-build-ros): build with RoboStack's build scripts - #7540
Tobias-Fischer wants to merge 4 commits into
Conversation
800c809 to
f1fe927
Compare
That seems a regression, ninja has many advantages w.r.t. visual studio. To avoid long paths failures we could try https://www.scivision.dev/cmake-short-build-path/ and keep ninja? |
f3c9d65 to
48847b4
Compare
Good point, thanks. pixi's template used Ninja, and the template already passes — Claude (on behalf of @Tobias-Fischer) |
…ges on Windows Forces a rebuild of builtin_interfaces (where the long-path failure appeared in RoboStack/ros-humble#229), control_msgs, nav2_msgs, rcl_interfaces and rosbag2_interfaces, to check bld_ament_cmake.bat with Ninja and CMAKE_INTERMEDIATE_DIR_STRATEGY=SHORT (prefix-dev/pixi#7540 review). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Follow-up: a RoboStack Windows build confirms Ninja works here. With — Claude (on behalf of @Tobias-Fischer) |
|
So it was just trying to be a good agent .. vinca has VS currently: https://github.com/RoboStack/vinca/blob/18ac8f5fc252bc819efc55633b8ac6e39078dd0a/vinca/templates/bld_ament_cmake.bat.in#L27 anyhow, fixed now :) |
823b80c to
15a59d8
Compare
vinca-rs from its templates-check branch (pixi-build-ros at prefix-dev/pixi#7540's unified ament_cmake/cmake templates, built from source), and rebuilds of ros_workspace, ament_cmake_test, rcutils, builtin_interfaces, fastcdr and ruckig on every platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vinca-rs from its templates-check branch (pixi-build-ros at prefix-dev/pixi#7540's unified ament_cmake/cmake templates, built from source), and rebuilds of ros_workspace, ament_cmake_test, rcutils, builtin_interfaces, fastcdr and ruckig on every platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
15a59d8 to
140a142
Compare
pixi-build-ros (prefix-dev/pixi#7540) now has one template per ROS 2 build type for every platform: ament_cmake and cmake share one, with ${CMAKE_ARGS}, and on Windows the recipe runs the bash script with brush (`script: {file, interpreter: brush}`, brush in the build requirements). The source directory is in bash syntax on Windows too, except for ROS 1's catkin batch file. A recipe whose script is a mapping counts as having a build script in the workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Results for the latest push, where ament_cmake and cmake share one template and Windows runs the ROS 2 bash scripts with brush:
— Claude |
vinca-rs with prefix-dev/pixi#7540's build scripts: ament_cmake and cmake share one template (with conda's ${CMAKE_ARGS}, BUILD_TESTING=OFF and a PYTHON_INSTALL_DIR relative to the prefix for both), and the ROS 2 scripts are bash scripts on every platform, run with brush on Windows. rtabmap's Windows link flags are in bash syntax ($LIBRARY_LIB). vinca-rs comes from its branch (a pixi source dependency) until 0.1.10 is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vinca-rs with prefix-dev/pixi#7540's build scripts: ament_cmake and cmake share one template (with conda's ${CMAKE_ARGS}, BUILD_TESTING=OFF and a PYTHON_INSTALL_DIR relative to the prefix for both), and the ROS 2 scripts are bash scripts on every platform, run with brush on Windows. rtabmap's Windows link flags are in bash syntax ($LIBRARY_LIB). vinca-rs comes from its branch (a pixi source dependency) until 0.1.10 is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
140a142 to
6506d3a
Compare
pixi-build-ros (prefix-dev/pixi#7540) now has one template per ROS 2 build type for every platform: ament_cmake and cmake share one, with ${CMAKE_ARGS}, and on Windows the recipe runs the bash script with brush (`script: {file, interpreter: brush}`, brush in the build requirements). The source directory is in bash syntax on Windows too, except for ROS 1's catkin batch file (pixi-build-ros's `is_batch` decides, as for the interpreter); the templates put it in double quotes, so it has no trailing separator (a batch file's `\"` would not close them). A recipe whose script is a mapping counts as having a build script in the workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vinca-rs with prefix-dev/pixi#7540's build scripts: ament_cmake and cmake share one template (with conda's ${CMAKE_ARGS}, BUILD_TESTING=OFF and a PYTHON_INSTALL_DIR relative to the prefix for both), and the ROS 2 scripts are bash scripts on every platform, run with brush on Windows. rtabmap's Windows link flags are in bash syntax ($LIBRARY_LIB). vinca-rs comes from its branch (a pixi source dependency) until 0.1.10 is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@Hofer-Julian this is ready for review (after #7539 ) |
The backend's templates were an old copy of RoboStack's, which have since
been fixed while building every package of RoboStack's distributions on five
platforms. Adopt those (they live here now, and RoboStack's recipe generator
renders them through this crate): among others the cross-compilation fixups,
the real `ROS_PYTHON_VERSION` (the old script exported `PY_VER=3`), conda's
macOS deployment target instead of a hardcoded 11.0, a `PYTHON_INSTALL_DIR`
that works without GNU realpath, short object paths on Windows
(`CMAKE_INTERMEDIATE_DIR_STRATEGY=SHORT`, so every Windows template builds with
Ninja; RoboStack's ament_cmake one had switched to Visual Studio for MSVC's
path limit).
ROS 2 cmake/catkin packages used the ROS 1 catkin template; they now share the
ament_cmake one, whose copies had drifted apart. The shared template takes
the ament_cmake settings for both (`BUILD_TESTING=OFF`, a `PYTHON_INSTALL_DIR`
relative to the prefix, as ament_cmake_python expects, and on Windows
`ROS_BUILD_SHARED_LIBS`/`NOGDI` for both) plus conda's `${CMAKE_ARGS}`, which
only the catkin-derived one passed. Dropped: `BOOST_ERROR_CODE_HEADER_ONLY` (a
no-op since Boost 1.69), `CMAKE_FIND_FRAMEWORK=LAST` (in conda's
`${CMAKE_ARGS}` on macOS), an always-empty `CMAKE_AUTOMOC_EXECUTABLE` and
`CMAKE_NO_SYSTEM_FROM_IMPORTED`. The catkin templates use conda's macOS
deployment target too.
Special cases by package name are gone where they are no longer needed: the
emscripten source directories (vinca-rs puts an additional recipe's folder in
`source_dir`), eigenpy's single-job Windows build (from 2021's 7 GB runners),
and the activation scripts of ros_workspace / catkin, which are now whatever
activation scripts a recipe ships. ROS 1's catkin cookie, `environment`
profile scripts and Makefile generator for euslisp stay.
What differs between `pixi build` and a RoboStack recipe is a parameter:
- `ScriptVars::source_dir`: the manifest directory here, the checked-out source
in a recipe (the templates no longer set `SRC_DIR`)
- `BuildMode::Incremental` keeps the build directory (the configure step is
incremental too, so the `build.ninja` guard is gone) and lists what the
build installed in `$RATTLER_BUILD_PACKAGE_FILES`, as before;
`BuildMode::Clean` is RoboStack's fresh build
- `ScriptVars::generator`: the scripts' first line names what wrote them
(`pixi-build-ros` here)
The catkin templates stay for ROS 1, with `@(package_prefix)-` instead of the
hardcoded `ros-noetic-` names. `BuildType::of` decides the template (no build
type plus an ament_cmake build tool is ament_cmake; catkin/cmake is the cmake
template in ROS 2 and the catkin one in ROS 1).
The templates put the source directory in double quotes, and `pixi build`
escapes the manifest directory for them (`escape_quoted`), so a path with spaces,
`$` or `%` works. Which template a package gets follows the host platform's
operating system (it was the machine generating the recipe, which made the
recipe depend on where it was generated; `pixi build` does not cross-compile
between operating systems). The gcc/g++ links on Linux are forced, so a
rebuild in the same build prefix works. The templates are left out of shfmt:
their `@[...]` line prefixes are not shell.
Tests: `test_every_template_is_filled` renders every template in both modes
for Unix and Windows; `test_build_type` covers the template choice;
`test_escape_quoted` the escaping.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…brush on Windows The ROS 2 templates were a bash script and a batch file per build type, kept in sync by hand. Now each build type has one bash script for every platform: on Windows the recipe runs it with brush (https://github.com/reubeno/brush, a bash-compatible shell; rattler-build runs the compiler activation with cmd.exe first), and `brush` is a build requirement there (`script_interpreter`). ROS 1's catkin template keeps its batch file (`is_batch`). The Windows differences are branches in the scripts: the `%LIBRARY_PREFIX%` layout, MSVC (`CC`/`CXX=cl.exe`, `CL` for `ROS_BUILD_SHARED_LIBS`/`NOGDI`), short object paths, and on Windows the ament_python script installs with `setup.py` into `%LIBRARY_PREFIX%`. brush has no coreutils builtins, so file operations use `cmake -E` and bash builtins, and `PYTHON_INSTALL_DIR` is computed with Python's `os.path.relpath` on every platform. brush 0.4.0 exits a script when an assignment follows a failed condition under `-e` and `-x` (fixed upstream in reubeno/brush#1248, not yet released), so the scripts turn off `-x` under brush. The brush requirement and the interpreter follow the same platform as the template, the host platform's operating system. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mplates A Windows checkout with autocrlf embedded them with CRLF, which brush rejects (syntax error), failing every ROS build on Windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2e74ca4 to
3148f5b
Compare
|
@Hofer-Julian - this is ready for review now that #7539 has been merged :) |
… scripts build_catkin.sh and bld_catkin.bat never used @(cmake_args), so a package's additional CMake arguments (vinca-rs's additional_cmake_args) were dropped for ROS 1 packages. Add them, and check in the template test that every CMake build type gets them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 2d90277db29b13ad7638f82ae1978cff79da0ed8)
vinca-rs with prefix-dev/pixi#7540's build scripts: ament_cmake and cmake share one template (with conda's ${CMAKE_ARGS}, BUILD_TESTING=OFF and a PYTHON_INSTALL_DIR relative to the prefix for both), and the ROS 2 scripts are bash scripts on every platform, run with brush on Windows. rtabmap's Windows link flags are in bash syntax ($LIBRARY_LIB). vinca-rs comes from its branch (a pixi source dependency) until 0.1.10 is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vinca-rs with prefix-dev/pixi#7540's build scripts: ament_cmake and cmake share one template (with conda's ${CMAKE_ARGS}, BUILD_TESTING=OFF and a PYTHON_INSTALL_DIR relative to the prefix for both), and the ROS 2 scripts are bash scripts on every platform, run with brush on Windows. rtabmap's Windows link flags are in bash syntax ($LIBRARY_LIB). vinca-rs comes from its branch (a pixi source dependency) until 0.1.10 is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Description
Part 2 of 3, split out of #7535 for easier review. It builds on #7539 (names; the scripts use its package prefix), so the first commit here is #7539 and the last two are this PR: "build with RoboStack's build scripts" adopts the scripts, and "one bash build script per build type, run with brush on Windows" replaces the Windows batch files. Each can be reviewed on its own. (GitHub's stacked pull requests don't support branches from forks, so this is a plain PR against main.)
Tobi human description
The pixi-build-ros build scripts were taken from a VERY old RoboStack/vinca version. There have been many improvements since in the build scripts, and this PR simply copies over these build scripts so that the new vinca-rs can benefit from it and doesn't need to duplicate build scripts (which is the status quo).
Another cool feature is that we now use brush to unify the build scripts between Linux and Windows, which again streamlines things substantially and will maintenance a lot easier in the future.
Lots of details in the AI summary if needed/wanted.
AI-generated details (Claude Opus 5.5)
The backend's build scripts were an old copy of RoboStack's. RoboStack's have since been fixed while building every package of its distributions on five platforms. This PR adopts them, and RoboStack's recipe generator renders them through this crate, so
pixi buildand RoboStack's binaries build a package the same way. The old scripts here were missing several of those fixes:PY_VER=3, overriding rattler-build's;ROS_PYTHON_VERSION=3and macOS 11.0 instead of conda-forge's deployment target;CMAKE_INTERMEDIATE_DIR_STRATEGY=SHORT(CMake ≥ 4.2) shortens object paths, so every template keeps Ninja.ROS 2
cmakepackages used the ROS 1 catkin script. Now they share the ament_cmake script, whose copies had drifted apart. Both get its settings (BUILD_TESTING=OFF, aPYTHON_INSTALL_DIRrelative to the prefix as ament_cmake_python expects, andROS_BUILD_SHARED_LIBS/NOGDIon Windows) plus conda's${CMAKE_ARGS}. Settings that did nothing or repeated${CMAKE_ARGS}are dropped:BOOST_ERROR_CODE_HEADER_ONLY(a no-op since Boost 1.69),CMAKE_FIND_FRAMEWORK=LAST, an always-emptyCMAKE_AUTOMOC_EXECUTABLE, andCMAKE_NO_SYSTEM_FROM_IMPORTED. Package-name special cases that are no longer needed are gone: the emscripten source directories, eigenpy's single-job Windows build, and the activation scripts of ros_workspace and catkin, which are now whatever activation scripts a recipe ships. ROS 1 keeps the catkin scripts, which now use conda's macOS deployment target. Built locally on osx-arm64 with the old and the new scripts, ros_workspace, ament_cmake_test, rcutils, builtin_interfaces, fastcdr and ruckig contain the same files and have the same run dependencies.One script for every platform. The ROS 2 build types have one bash script each, also on Windows: the recipe runs it with brush, a bash-compatible shell, which rattler-build supports as
interpreter: brushafter setting up the compiler environment with cmd.exe.brushgoes into the Windows build requirements (script_interpreter). The Windows differences are branches in the script: the%LIBRARY_PREFIX%layout, MSVC,CL, short object paths. File operations that also run on Windows usecmake -E. brush 0.4.0 exits after an assignment in anelsebranch when it runs with both-eand-x, so the scripts turn command tracing off under brush (fixed upstream in reubeno/brush#1248, not released yet). The catkin scripts keep their batch file for ROS 1. Locally on osx-arm64, the six packages above and ament_index_python and demo_nodes_py run under brush give the same packages as under bash.What differs between
pixi buildand a RoboStack recipe is a parameter:ScriptVarsholds the source directory, the package prefix, the distro, extra CMake arguments, and the name of the tool that wrote the script, which goes on its first line.BuildMode::Incrementalis whatpixi builduses: the build directory is kept and the installed files are recorded in$RATTLER_BUILD_PACKAGE_FILES, as before. CMake now reconfigures on every build, which is fast on a kept directory.BuildMode::Cleanis RoboStack's fresh build.Kept for RoboStack and inert for
pixi build: the opt-inVINCA_USE_SCCACHEandVINCA_CUSTOM_CMAKE_BUILD_DIRenvironment variables, which RoboStack's CI sets.Also fixed along the way:
The templates put the source directory in double quotes, and
pixi buildescapes the manifest path for them, so paths with spaces,$or%work.Which script a package gets (and brush) follows the host platform's operating system, not the machine that generates the recipe, so a recipe no longer depends on where it was generated.
The gcc/g++ links on Linux are forced, so a rebuild in the kept build prefix works.
The templates are excluded from shfmt: their
@[...]line prefixes are not shell.Tests:
test_every_template_is_filledrenders every template in both modes for Unix and Windows and checks that no placeholder is left.test_build_typecovers which template is chosen. There are no snapshots of the rendered scripts; they would repeat the templates.ros_workspace's activation scripts, which feat(pixi-build-ros): RoboStack's build scripts, requirement rules and package names #7535 also moved here, stay in RoboStack's generator becausepixi builddoesn't use them.Checklist:
🤖 Generated with Claude Code