Skip to content

ci: build protobuf from source for the macOS wheels - #50

Merged
trendvidia merged 3 commits into
mainfrom
ci/macos-protobuf-from-source
Nov 8, 2026
Merged

trendvidia merged 3 commits into
mainfrom
ci/macos-protobuf-from-source

Conversation

@bfrancojr

Copy link
Copy Markdown
Collaborator

Fixes #20.

What changes

The macOS wheel's support floor (macOS 15.0) no longer depends on which macOS the CI runner has. protobuf and abseil are now built from source at MACOSX_DEPLOYMENT_TARGET and linked statically, so delocate has no Homebrew dylib to compare against the wheel's target. The wheel-building runners go back to macos-latest in ci.yml and publish.yml. The floor itself is unchanged.

scripts/cibw_install_protobuf_linux.sh becomes scripts/cibw_install_protobuf.sh, used by both OSes. On macOS it differs in three ways, each found by building locally:

  • protobuf v36.2 instead of v25.3. v25.3's bundled abseil (LTS 20230802) fails on Apple arm64. It passes -Xarch_x86_64 -msse4.1 as two options, CMake drops the repeated -Xarch_x86_64, and clang refuses the bare -msse4.1. The failing target's flags.make shows -Xarch_x86_64 -maes -msse4.1. v36.2 fetches abseil 20250512.1, which has the fix. It's also the version Homebrew ships, so the macOS test legs already build against it.
  • abseil always fetched (protobuf_FORCE_FETCH_DEPENDENCIES=ON), never taken from Homebrew.
  • CMAKE_CXX_STANDARD=17. Without it the installed abseil uses its own absl::string_view rather than std::string_view, and protowire-cpp fails to compile (no viable conversion from 'absl::string_view' to 'std::string_view' in encode.cc and wellknown.cc).

The script refuses to run on macOS without MACOSX_DEPLOYMENT_TARGET, so it can't quietly build for the runner's macOS again.

Linux stays on v25.3. This PR's checks build only the Linux x86_64 wheel. The aarch64 one is built only at release, so a version change there couldn't be proven here.

Evidence (local, macOS 27.0.1 arm64, Apple clang 21)

This host's macOS is newer than the 15.0 target, which is the case that broke in #18. It also has Homebrew's protobuf/abseil installed and a stale /usr/local/include/google/protobuf, a dirtier setup than the runner.

  • Script with MACOSX_DEPLOYMENT_TARGET=15.0: exit 0, libprotoc 36.2. 94 static libraries, 0 dylibs. minos 15.0 on protoc and on the sampled objects in libprotobuf.a and abseil's randen_hwaes. Log: "Fallback to downloading Abseil 20250512.1".
  • pip wheel with CMAKE_PREFIX_PATH at that prefix: CMake reports Found Protobuf: <prefix>/lib/libprotobuf.a. Wheel: protowire_python-1.1.0-cp315-cp315-macosx_15_0_arm64.whl.
  • delocate-wheel --require-archs arm64: exit 0. delocate-listdeps --all lists only system libraries (CoreFoundation, libSystem, libc++, libz). otool -l on _protowire.so: minos 15.0.
  • Wheel installed into a fresh venv, then pytest tests: 200 passed. A probe test printed _protowire loading from the venv's site-packages, not from src/.

Not yet shown: the macos-latest cibuildwheel leg on GitHub, the last done-when item of #20. This PR's cibuildwheel smoke / macos-latest job is that check.

The macOS wheel's support floor was set by the runner image: Homebrew's
protobuf/abseil bottles are built for the runner's macOS, and delocate
refuses a dylib whose minimum target is newer than the wheel's. That
broke the wheel when macos-latest moved to macOS 26 (#18), and #19 fixed
it by pinning the runner to macos-15.

Build protobuf and abseil from source at MACOSX_DEPLOYMENT_TARGET and
link them statically, as the Linux leg already does. delocate then has
no dylib to check, and the runners go back to macos-latest.

The Linux script becomes scripts/cibw_install_protobuf.sh for both OSes.
macOS needs three differences, each found by building on macOS 27:
- protobuf v36.2, not v25.3. v25.3 bundles abseil LTS 20230802, which
  passes "-Xarch_x86_64 -msse4.1" as separate options; CMake drops the
  repeated -Xarch_x86_64 and clang refuses the bare -msse4.1 on arm64.
  Abseil 20240722 fixed it. v36.2 is what Homebrew ships, so the macOS
  test legs already build against it.
- protobuf_FORCE_FETCH_DEPENDENCIES=ON, so abseil is never Homebrew's
  dylib.
- CMAKE_CXX_STANDARD=17. Abseil pins absl::string_view to the std type
  only if a test compile at the default standard is C++17; Apple clang's
  default is older, and protowire-cpp then fails to compile.

Linux stays on v25.3: the PR's checks build only its x86_64 leg, not the
aarch64 one the release builds.

Fixes #20
The smoke log showed the extension built as macosx_11_0 (cibuildwheel's
default for arm64) and retagged macosx_15_0 by delocate only because the
linked protobuf code targets 15.0. Main's last run shows the same. The
floor came out right, but by way of delocate rather than the setting.
Pass MACOSX_DEPLOYMENT_TARGET through CIBW_ENVIRONMENT_MACOS so the build
itself targets it.
This reverts commit c0002de.

The cause that commit gave was wrong. cibuildwheel already passes the
job's MACOSX_DEPLOYMENT_TARGET to the build (env.setdefault,
cibuildwheel/platforms/macos.py:360 at v4.3.5). The macosx_11_0 name in
the log comes from _PYTHON_HOST_PLATFORM, which cibuildwheel sets to
macosx-11.0-arm64 for every arm64 build (line 372). scikit-build-core
names the wheel from it, and delocate renames it from the binaries' real
target. The re-run with the change still showed "Created ...
macosx_11_0", so it changed nothing.
@bfrancojr

Copy link
Copy Markdown
Collaborator Author

CI result at 95ee248: all 18 checks passed. That includes cibuildwheel smoke / macos-latest, on runner image macos-26-arm64. Its log shows protobuf built from source (libprotoc 36.2, "Fallback to downloading Abseil 20250512.1", Found Protobuf: /tmp/protobuf-install/lib/libprotobuf.a) and the repaired wheel protowire_python-1.1.0-cp312-cp312-macosx_15_0_arm64.whl. That's the last done-when item of #20.

A detour, now reverted. The log also showed the wheel first created as macosx_11_0, then renamed macosx_15_0 by delocate. Main's last run does the same. I guessed that the deployment target wasn't reaching the build, and pushed c0002de to pass it through CIBW_ENVIRONMENT_MACOS. The re-run still created macosx_11_0, so the guess was wrong. cibuildwheel v4.3.5 already passes the job's MACOSX_DEPLOYMENT_TARGET to the build (platforms/macos.py:360). The 11_0 name comes from _PYTHON_HOST_PLATFORM, which it sets to macosx-11.0-arm64 for every arm64 build (line 372), and delocate renames the wheel from the binaries' actual target. a60a6a3 reverts c0002de, so the tree is identical to 95ee248 (git diff 95ee248 a60a6a3 is empty).

@trendvidia
trendvidia merged commit ef0843d into main Nov 8, 2026
18 checks passed
@trendvidia
trendvidia deleted the ci/macos-protobuf-from-source branch November 8, 2026 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: decouple the wheel's macOS support floor from the runner image (build protobuf/absl from source)

2 participants