From 95ee2484ee17848d6357bdbe5b64e4d5d9705707 Mon Sep 17 00:00:00 2001 From: "Benedicto Franco Jr." Date: Sat, 7 Nov 2026 16:22:07 -0800 Subject: [PATCH 1/3] ci: build protobuf from source for the macOS wheels The macOS wheel's support floor was set by the runner image: Homebrew's protobuf/abseil bottles are built for the runner's macOS, and delocate refuses a dylib whose minimum target is newer than the wheel's. That broke the wheel when macos-latest moved to macOS 26 (#18), and #19 fixed it by pinning the runner to macos-15. Build protobuf and abseil from source at MACOSX_DEPLOYMENT_TARGET and link them statically, as the Linux leg already does. delocate then has no dylib to check, and the runners go back to macos-latest. The Linux script becomes scripts/cibw_install_protobuf.sh for both OSes. macOS needs three differences, each found by building on macOS 27: - protobuf v36.2, not v25.3. v25.3 bundles abseil LTS 20230802, which passes "-Xarch_x86_64 -msse4.1" as separate options; CMake drops the repeated -Xarch_x86_64 and clang refuses the bare -msse4.1 on arm64. Abseil 20240722 fixed it. v36.2 is what Homebrew ships, so the macOS test legs already build against it. - protobuf_FORCE_FETCH_DEPENDENCIES=ON, so abseil is never Homebrew's dylib. - CMAKE_CXX_STANDARD=17. Abseil pins absl::string_view to the std type only if a test compile at the default standard is C++17; Apple clang's default is older, and protowire-cpp then fails to compile. Linux stays on v25.3: the PR's checks build only its x86_64 leg, not the aarch64 one the release builds. Fixes #20 --- .github/workflows/ci.yml | 35 +++++------ .github/workflows/publish.yml | 37 ++++++------ CHANGELOG.md | 9 +++ scripts/cibw_install_protobuf.sh | 83 ++++++++++++++++++++++++++ scripts/cibw_install_protobuf_linux.sh | 42 ------------- 5 files changed, 126 insertions(+), 80 deletions(-) create mode 100755 scripts/cibw_install_protobuf.sh delete mode 100755 scripts/cibw_install_protobuf_linux.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8e6c341..4977dc3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -123,11 +123,11 @@ jobs: strategy: fail-fast: false matrix: - # macOS is pinned, not -latest: the wheel's deployment target and - # the runner's Homebrew bottles must agree. See MACOSX_DEPLOYMENT_TARGET - # below. The test matrix above stays on macos-latest — it installs - # editable and never runs delocate, so it is unaffected. - os: [ubuntu-latest, macos-15, windows-latest] + # macos-latest on purpose: its macOS is newer than the wheel's + # deployment target below, which is the case that broke in #18. + # A green leg here shows the target is set independently of the + # runner image. + os: [ubuntu-latest, macos-latest, windows-latest] steps: - uses: actions/checkout@v7 @@ -152,26 +152,21 @@ jobs: # container; host paths don't resolve. Override there. CIBW_ENVIRONMENT_LINUX: PROTOWIRE_CPP_DIR=/project/protowire-cpp-checkout PROTOWIRE_CPP_DIR: ${{ github.workspace }}/protowire-cpp-checkout - # Linux uses manylinux containers — base image protobuf is - # too old (3.5 / 3.14) for our annotations, so build 25.3 - # from source. See scripts/cibw_install_protobuf_linux.sh - # for the why. - CIBW_BEFORE_ALL_LINUX: bash {project}/scripts/cibw_install_protobuf_linux.sh - CIBW_BEFORE_ALL_MACOS: brew install protobuf + # Linux and macOS build protobuf 25.3 from source, as publish.yml + # does. See scripts/cibw_install_protobuf.sh for the why. + CIBW_BEFORE_ALL_LINUX: bash {project}/scripts/cibw_install_protobuf.sh + CIBW_BEFORE_ALL_MACOS: PROTOBUF_PREFIX=/tmp/protobuf-install bash {project}/scripts/cibw_install_protobuf.sh + CIBW_ENVIRONMENT_MACOS: CMAKE_PREFIX_PATH=/tmp/protobuf-install CIBW_BEFORE_ALL_WINDOWS: vcpkg install protobuf:x64-windows # delvewheel is only auto-installed when cibuildwheel uses # its default Windows repair command — once we override # CIBW_REPAIR_WHEEL_COMMAND_WINDOWS we have to install it. CIBW_BEFORE_BUILD_WINDOWS: pip install delvewheel - # macOS: Homebrew's bottled dylibs are built for the runner's OS, - # and delocate refuses to bundle dylibs whose min target is newer - # than the wheel's. So this value and the runner image are ONE - # decision: keep it equal to the pinned macos-15 in the matrix - # above. Do not switch that runner to macos-latest — GitHub rolled - # macos-latest to macOS 26 in July 2026, whose bottles target 26.0, - # which broke this job for six weeks (#18). Raising both together - # is a support-floor change: it drops every macOS user below the - # new value, so it is a release decision, not a CI tweak. + # The wheel's macOS support floor. protobuf is built from source + # at this target and linked statically, so the value does not + # depend on the runner image (#20). Changing it drops or adds + # macOS users, so it is a release decision, not a CI tweak; keep + # it equal to publish.yml's macosx_deployment_target. MACOSX_DEPLOYMENT_TARGET: "15.0" CIBW_ENVIRONMENT_WINDOWS: >- CMAKE_TOOLCHAIN_FILE="$VCPKG_INSTALLATION_ROOT/scripts/buildsystems/vcpkg.cmake" diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index f7367e0..ff5ba0a 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -52,17 +52,14 @@ jobs: # macOS users fall back to a source build via the published # sdist for now; we'll add the x86_64 wheel back via # universal2 (or a self-hosted runner) in a follow-up. - # The runner image and macosx_deployment_target below are ONE - # decision, not two: Homebrew's bottled protobuf/abseil dylibs are - # built for the runner's macOS version, and delocate refuses to - # bundle dylibs whose min target is newer than the wheel's. So the - # runner is pinned rather than -latest. GitHub rolled macos-latest - # to macOS 26 in July 2026, whose bottles target 26.0, silently - # breaking the macOS wheel against the 15.0 target here (#18). - # Raising this pair sets the published wheel's macOS support floor - # — it drops every macOS user below the new value — so it is a - # release decision. Keep the two values equal. - - os: macos-15 + # macosx_deployment_target is the published wheel's macOS support + # floor, and it no longer depends on the runner image: protobuf and + # abseil are built from source at this target and linked statically + # (scripts/cibw_install_protobuf.sh), so delocate has no Homebrew + # dylib to compare against it (#18, #20). The runner can move + # freely. Changing this value drops or adds macOS users, so it is a + # release decision. + - os: macos-latest archs: "arm64" macosx_deployment_target: "15.0" - os: windows-latest @@ -99,11 +96,15 @@ jobs: # macOS + Windows: cibuildwheel runs natively, so the host # path passed via the action-level env block works directly. PROTOWIRE_CPP_DIR: ${{ github.workspace }}/protowire-cpp-checkout - # The manylinux base image's protobuf-devel rpm is too old - # (3.5 / 3.14) and generates broken C++ for fields named - # `default`. Build 25.3 from source instead. - CIBW_BEFORE_ALL_LINUX: bash {project}/scripts/cibw_install_protobuf_linux.sh - CIBW_BEFORE_ALL_MACOS: brew install protobuf + # Build protobuf 25.3 from source on Linux and macOS. Linux: the + # manylinux base image's protobuf-devel rpm is too old (3.5 / 3.14) + # and generates broken C++ for fields named `default`. macOS: built + # at MACOSX_DEPLOYMENT_TARGET so the wheel's floor is not the + # runner's (#20). See scripts/cibw_install_protobuf.sh. + CIBW_BEFORE_ALL_LINUX: bash {project}/scripts/cibw_install_protobuf.sh + CIBW_BEFORE_ALL_MACOS: PROTOBUF_PREFIX=/tmp/protobuf-install bash {project}/scripts/cibw_install_protobuf.sh + # The build finds that install before anything Homebrew has. + CIBW_ENVIRONMENT_MACOS: CMAKE_PREFIX_PATH=/tmp/protobuf-install # Use the default dynamic triplet — vcpkg integrates with # CMake's find_package via the toolchain file, and delvewheel # bundles the resulting libprotobuf.dll / libabsl_*.dll into @@ -117,8 +118,8 @@ jobs: # CIBW_REPAIR_WHEEL_COMMAND_WINDOWS we have to install it # ourselves. CIBW_BEFORE_BUILD_WINDOWS: pip install delvewheel - # See matrix comment: align wheel target with runner OS so - # delocate accepts brew's dylibs. + # The wheel's macOS floor; see the matrix comment. It is set + # independently of the runner image. MACOSX_DEPLOYMENT_TARGET: ${{ matrix.macosx_deployment_target }} CIBW_ENVIRONMENT_WINDOWS: >- CMAKE_TOOLCHAIN_FILE="$VCPKG_INSTALLATION_ROOT/scripts/buildsystems/vcpkg.cmake" diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f73c12..8021d33 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -144,6 +144,15 @@ format changes. the corpus's 101-level rows never reached the encoder. A read that fails still exits 2. +- **The macOS wheel's support floor no longer depends on the CI runner's + macOS** (#20). protobuf and abseil are now built from source at + `MACOSX_DEPLOYMENT_TARGET` and linked statically + (`scripts/cibw_install_protobuf.sh`, which replaces + `cibw_install_protobuf_linux.sh`), instead of taken from Homebrew bottles + built for the runner. The wheel-building runners are back on + `macos-latest`. The floor is unchanged at macOS 15.0. macOS builds + protobuf v36.2; Linux stays on v25.3. + ## [1.1.0] — 2026-10-31 ### Added diff --git a/scripts/cibw_install_protobuf.sh b/scripts/cibw_install_protobuf.sh new file mode 100755 index 0000000..f4743bf --- /dev/null +++ b/scripts/cibw_install_protobuf.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MIT +# Copyright (c) 2026 TrendVidia, LLC. +# +# Build protobuf (and its bundled abseil) from source for a cibuildwheel +# leg, as static PIC libraries that the native module absorbs. +# +# Linux: runs inside the manylinux container. The base image's +# protobuf-devel rpm is too old (AlmaLinux 8 → 3.5, AlmaLinux 9 → 3.14). +# Protoc only started escaping C++ reserved-word field names (e.g. +# `default` in pxf/annotations.proto) in 3.21, so 3.x rpms generate +# broken `.pb.cc` for our annotations. +# +# macOS: runs on the runner itself. Homebrew's bottles are built for the +# runner's macOS version, and delocate refuses to bundle a dylib whose +# minimum target is newer than the wheel's, so with bottles the wheel's +# support floor was tied to the runner image (#18, #20). Built here at +# MACOSX_DEPLOYMENT_TARGET and linked statically, there is no dylib for +# delocate to check, and the floor is whatever MACOSX_DEPLOYMENT_TARGET +# says. The libraries are built for the runner's architecture. +# +# Versions differ by OS: +# - Linux: v25.3, last stable in the v25 line, before v26's abseil API +# churn. Its bundled abseil is a git submodule. +# - macOS: v36.2, the version Homebrew ships, which the macOS test legs +# already build against. v25.3 cannot build here: its bundled abseil +# (LTS 20230802) passes `-Xarch_x86_64 -msse4.1` as two separate +# options, CMake drops the repeated `-Xarch_x86_64`, and clang refuses +# the bare `-msse4.1` on arm64. Abseil 20240722 fixed it; protobuf v30+ +# fetches a fixed abseil (20250512.1 for v36.2) instead of bundling one. +# It is always fetched, never taken from Homebrew, whose abseil is a +# dylib built for the runner's macOS. +# +# Installs into PROTOBUF_PREFIX (default /usr/local). A prefix other than +# a system one must also be on the build's CMAKE_PREFIX_PATH. + +set -euxo pipefail + +PROTOBUF_VERSION=25.3 +PREFIX="${PROTOBUF_PREFIX:-/usr/local}" +WORKDIR="${TMPDIR:-/tmp}/protobuf-build" + +cmake_args=() +if [[ "$(uname -s)" == Darwin ]]; then + PROTOBUF_VERSION=36.2 + cmake_args+=("-Dprotobuf_FORCE_FETCH_DEPENDENCIES=ON") + # Abseil pins absl::string_view to std::string_view at install time only + # if a test compile at the default standard is C++17 or later. Apple + # clang's default is older, so without this the installed abseil uses + # its own string_view and protowire-cpp fails to compile against it. + # Homebrew builds abseil at 17 too. + cmake_args+=("-DCMAKE_CXX_STANDARD=17") + # The deployment target is the wheel's support floor; building without + # one would quietly target the runner's own macOS again. + : "${MACOSX_DEPLOYMENT_TARGET:?set MACOSX_DEPLOYMENT_TARGET to the macOS floor of the wheel}" + cmake_args+=("-DCMAKE_OSX_DEPLOYMENT_TARGET=${MACOSX_DEPLOYMENT_TARGET}") +fi + +# The release-page tarball excludes git submodules (abseil-cpp, +# utf8_range), so a curl|tar pipeline gives an empty third_party/. +# Use a shallow clone with --recurse-submodules instead. +rm -rf "$WORKDIR" +git clone --depth 1 --branch "v${PROTOBUF_VERSION}" \ + --recurse-submodules --shallow-submodules \ + https://github.com/protocolbuffers/protobuf.git "$WORKDIR" +cd "$WORKDIR" + +cmake -B build \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_POSITION_INDEPENDENT_CODE=ON \ + -Dprotobuf_BUILD_TESTS=OFF \ + -Dprotobuf_BUILD_SHARED_LIBS=OFF \ + -Dprotobuf_ABSL_PROVIDER=module \ + -DCMAKE_INSTALL_PREFIX="$PREFIX" \ + ${cmake_args[@]+"${cmake_args[@]}"} +cmake --build build -j"$(getconf _NPROCESSORS_ONLN)" +cmake --install build +if [[ "$(uname -s)" == Linux ]]; then + ldconfig 2>/dev/null || true +fi + +# Sanity check: the installed protoc must be 3.21+. +"$PREFIX/bin/protoc" --version diff --git a/scripts/cibw_install_protobuf_linux.sh b/scripts/cibw_install_protobuf_linux.sh deleted file mode 100755 index b751702..0000000 --- a/scripts/cibw_install_protobuf_linux.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/usr/bin/env bash -# SPDX-License-Identifier: MIT -# Copyright (c) 2026 TrendVidia, LLC. -# -# Install a recent protobuf inside the cibuildwheel manylinux container. -# -# Why: the base image's protobuf-devel rpm is too old (AlmaLinux 8 → -# 3.5, AlmaLinux 9 → 3.14). Protoc only started escaping C++ -# reserved-word field names (e.g. `default` in pxf/annotations.proto) -# in 3.21, so 3.x rpms generate broken `.pb.cc` for our annotations. -# -# Pinned to v25.3 — last stable in the v25 line, before v26's abseil -# API churn. Static + PIC so the eventual nanobind shared module can -# absorb it without ldconfig drift. - -set -euxo pipefail - -PROTOBUF_VERSION=25.3 -WORKDIR=/tmp/protobuf-build - -# The release-page tarball excludes git submodules (abseil-cpp, -# utf8_range), so a curl|tar pipeline gives an empty third_party/. -# Use a shallow clone with --recurse-submodules instead. -rm -rf "$WORKDIR" -git clone --depth 1 --branch "v${PROTOBUF_VERSION}" \ - --recurse-submodules --shallow-submodules \ - https://github.com/protocolbuffers/protobuf.git "$WORKDIR" -cd "$WORKDIR" - -cmake -B build \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_POSITION_INDEPENDENT_CODE=ON \ - -Dprotobuf_BUILD_TESTS=OFF \ - -Dprotobuf_BUILD_SHARED_LIBS=OFF \ - -Dprotobuf_ABSL_PROVIDER=module \ - -DCMAKE_INSTALL_PREFIX=/usr/local -cmake --build build -j"$(nproc)" -cmake --install build -ldconfig 2>/dev/null || true - -# Sanity check: the installed protoc must be 3.21+. -/usr/local/bin/protoc --version From c0002decf34312e939d21c458954e0d99bdb1ac8 Mon Sep 17 00:00:00 2001 From: "Benedicto Franco Jr." Date: Sat, 7 Nov 2026 16:51:44 -0800 Subject: [PATCH 2/3] ci: pass the macOS deployment target into cibuildwheel's build The smoke log showed the extension built as macosx_11_0 (cibuildwheel's default for arm64) and retagged macosx_15_0 by delocate only because the linked protobuf code targets 15.0. Main's last run shows the same. The floor came out right, but by way of delocate rather than the setting. Pass MACOSX_DEPLOYMENT_TARGET through CIBW_ENVIRONMENT_MACOS so the build itself targets it. --- .github/workflows/ci.yml | 7 ++++++- .github/workflows/publish.yml | 7 ++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4977dc3..4a6d448 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -156,7 +156,12 @@ jobs: # does. See scripts/cibw_install_protobuf.sh for the why. CIBW_BEFORE_ALL_LINUX: bash {project}/scripts/cibw_install_protobuf.sh CIBW_BEFORE_ALL_MACOS: PROTOBUF_PREFIX=/tmp/protobuf-install bash {project}/scripts/cibw_install_protobuf.sh - CIBW_ENVIRONMENT_MACOS: CMAKE_PREFIX_PATH=/tmp/protobuf-install + # cibuildwheel builds the extension at its own default target (11.0) + # unless MACOSX_DEPLOYMENT_TARGET is passed here; the step env alone + # does not reach the build. + CIBW_ENVIRONMENT_MACOS: >- + CMAKE_PREFIX_PATH=/tmp/protobuf-install + MACOSX_DEPLOYMENT_TARGET=15.0 CIBW_BEFORE_ALL_WINDOWS: vcpkg install protobuf:x64-windows # delvewheel is only auto-installed when cibuildwheel uses # its default Windows repair command — once we override diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index ff5ba0a..118ddc3 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -104,7 +104,12 @@ jobs: CIBW_BEFORE_ALL_LINUX: bash {project}/scripts/cibw_install_protobuf.sh CIBW_BEFORE_ALL_MACOS: PROTOBUF_PREFIX=/tmp/protobuf-install bash {project}/scripts/cibw_install_protobuf.sh # The build finds that install before anything Homebrew has. - CIBW_ENVIRONMENT_MACOS: CMAKE_PREFIX_PATH=/tmp/protobuf-install + # cibuildwheel builds the extension at its own default target (11.0) + # unless MACOSX_DEPLOYMENT_TARGET is passed here; the step env alone + # does not reach the build. + CIBW_ENVIRONMENT_MACOS: >- + CMAKE_PREFIX_PATH=/tmp/protobuf-install + MACOSX_DEPLOYMENT_TARGET=${{ matrix.macosx_deployment_target }} # Use the default dynamic triplet — vcpkg integrates with # CMake's find_package via the toolchain file, and delvewheel # bundles the resulting libprotobuf.dll / libabsl_*.dll into From a60a6a3d65c98fd5274080a3c16ca63acb938cb9 Mon Sep 17 00:00:00 2001 From: "Benedicto Franco Jr." Date: Sat, 7 Nov 2026 17:00:47 -0800 Subject: [PATCH 3/3] Revert "ci: pass the macOS deployment target into cibuildwheel's build" This reverts commit c0002decf34312e939d21c458954e0d99bdb1ac8. The cause that commit gave was wrong. cibuildwheel already passes the job's MACOSX_DEPLOYMENT_TARGET to the build (env.setdefault, cibuildwheel/platforms/macos.py:360 at v4.3.5). The macosx_11_0 name in the log comes from _PYTHON_HOST_PLATFORM, which cibuildwheel sets to macosx-11.0-arm64 for every arm64 build (line 372). scikit-build-core names the wheel from it, and delocate renames it from the binaries' real target. The re-run with the change still showed "Created ... macosx_11_0", so it changed nothing. --- .github/workflows/ci.yml | 7 +------ .github/workflows/publish.yml | 7 +------ 2 files changed, 2 insertions(+), 12 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4a6d448..4977dc3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -156,12 +156,7 @@ jobs: # does. See scripts/cibw_install_protobuf.sh for the why. CIBW_BEFORE_ALL_LINUX: bash {project}/scripts/cibw_install_protobuf.sh CIBW_BEFORE_ALL_MACOS: PROTOBUF_PREFIX=/tmp/protobuf-install bash {project}/scripts/cibw_install_protobuf.sh - # cibuildwheel builds the extension at its own default target (11.0) - # unless MACOSX_DEPLOYMENT_TARGET is passed here; the step env alone - # does not reach the build. - CIBW_ENVIRONMENT_MACOS: >- - CMAKE_PREFIX_PATH=/tmp/protobuf-install - MACOSX_DEPLOYMENT_TARGET=15.0 + CIBW_ENVIRONMENT_MACOS: CMAKE_PREFIX_PATH=/tmp/protobuf-install CIBW_BEFORE_ALL_WINDOWS: vcpkg install protobuf:x64-windows # delvewheel is only auto-installed when cibuildwheel uses # its default Windows repair command — once we override diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 118ddc3..ff5ba0a 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -104,12 +104,7 @@ jobs: CIBW_BEFORE_ALL_LINUX: bash {project}/scripts/cibw_install_protobuf.sh CIBW_BEFORE_ALL_MACOS: PROTOBUF_PREFIX=/tmp/protobuf-install bash {project}/scripts/cibw_install_protobuf.sh # The build finds that install before anything Homebrew has. - # cibuildwheel builds the extension at its own default target (11.0) - # unless MACOSX_DEPLOYMENT_TARGET is passed here; the step env alone - # does not reach the build. - CIBW_ENVIRONMENT_MACOS: >- - CMAKE_PREFIX_PATH=/tmp/protobuf-install - MACOSX_DEPLOYMENT_TARGET=${{ matrix.macosx_deployment_target }} + CIBW_ENVIRONMENT_MACOS: CMAKE_PREFIX_PATH=/tmp/protobuf-install # Use the default dynamic triplet — vcpkg integrates with # CMake's find_package via the toolchain file, and delvewheel # bundles the resulting libprotobuf.dll / libabsl_*.dll into