Add ppc64le power9, power10, and power11 optimized build targets - #1220
Add ppc64le power9, power10, and power11 optimized build targets#1220veenious wants to merge 1 commit into
Conversation
|
As mentioned in #1215 before considering adding these targets I would like to see benchmarks which should that there is a benefit to building this microarchitecture specific distributions. Additional the change to bookworm likely raises the minimum glibc version required to consume these distributions from 2.24 (Debian 9 Stretch) to 2.36 (Debian 12 Bookworm). A better approach that is more maintainable in the long term to to use the LLVM toolchain to cross-compile these targets using a sysroot. This was mentioned in #1215. |
ec991c1 to
a61a7be
Compare
Adds three new ppc64le architecture variant targets, mirroring the existing x86_64_v2/v3/v4 pattern: - ppc64le_power9-unknown-linux-gnu — compiled with -mcpu=power9 -mtune=power9 - ppc64le_power10-unknown-linux-gnu — compiled with -mcpu=power10 -mtune=power10 - ppc64le_power11-unknown-linux-gnu — compiled with -mcpu=power11 -mtune=power11 The existing generic ppc64le-unknown-linux-gnu artifact is untouched. Changes: - ci-targets.yaml: add all three power variant CI targets - cpython-unix/targets.yml: add build definitions with correct compiler paths; power9 reuses build.cross.Dockerfile (Debian Stretch + GCC 6), power10 uses new Bookworm + GCC 12 image, power11 uses new Bookworm + GCC 14 from Trixie image (GCC 12 does not support -mcpu=power11) - cpython-unix/build.cross.Dockerfile: switch apt sources to archive.debian.org over HTTPS (port 80 blocked in build containers) - cpython-unix/build.cross-ppc64le_power10.Dockerfile: new image - cpython-unix/build.cross-ppc64le_power11.Dockerfile: new image - cpython-unix/build.py: strip ppc64le_powerN- variant prefix in triple normalization, matching the x86_64_vN pattern - pythonbuild/downloads.json: fix x.org and sourceforge download URLs that were unreachable (redirect to blocked port 80) - src/release.rs: register all three triples in RELEASE_TRIPLES - src/validation.rs: add RECOGNIZED_TRIPLES, GLIBC_MAX_VERSION, ELF_ALLOWED_LIBRARIES, PLATFORM_TAG, and ELF machine entries Power9 support is based on Christy Norman's work (clnperez/ppc64le-currency). Power11 requires GCC 14 because GCC 12 does not support -mcpu=power11. Tested by cross-compiling on x86 and running on physical POWER9, POWER10, and POWER11 hardware (RHEL 9.8, glibc 2.34, UBI 10 container glibc 2.39). Closes astral-sh#1215
a61a7be to
cbcc593
Compare
Hardware validation + apples-to-apples benchmark — Python 3.13.15 on real P9/P10/P11 hardwareAll three binaries were cross-compiled from x86 (commit cbcc593) and tested on RHEL 9.8 machines inside Smoke tests — ALL PASSEDAll three binaries passed: Python version check, Performance benchmark — generic vs tuned, same machineTo isolate the tuning benefit from hardware differences, the same generic Workload: Pi to 10,000 digits via Python's POWER10
→ 38.7% faster on the same P10 machine with POWER11
→ 38.3% faster on the same P11 machine with glibc requirementPower9 requires glibc ≥ 2.17 (same as the generic baseline). Power10 and Power11 require glibc ≥ 2.36 due to the Bookworm build base. This is enforced in |
| # Stretch stopped publishing snapshots in April 2023. Last snapshot | ||
| # is 20230423T032533Z. But there are package authentication issues | ||
| # with this snapshot. | ||
| RUN for s in debian_stretch debian_stretch-updates debian-security_stretch/updates; do \ | ||
| echo "deb http://snapshot.debian.org/archive/${s%_*}/20221105T150728Z/ ${s#*_} main"; \ | ||
| done > /etc/apt/sources.list && \ |
There was a problem hiding this comment.
The snapshot release should not be removed. We want reproducible images when possible
|
A good amount of work is done to keep the x86-64 microarchitecture variants based on the same glibc version. The same should be done here. There is limited functionality in uv to select a build based on the host glibc version so supporting older versions is important. I would appreciate more details benchmarks, at minimum using fastmark but if possible pyperformance. These would give a more complete picture. The benchmarks should be based on the same compiler version as these difference can be substantial. If upgrading the compiler (and glibc version) give the bulk of the performance gain there is no reason to expand the build matrix. Is there a reason why cross-compiling using LLVM was not been tried discussed in #1215 and my earlier comment? |
| @@ -0,0 +1,54 @@ | |||
| # Debian Bookworm. | |||
| FROM docker.io/library/debian@sha256:6bc30d909583f38600edd6609e29eb3fb284ab8affce8d0389f332fc91c2dd91 | |||
| LABEL org.opencontainers.image.authors="Gregory Szorc <gregory.szorc@gmail.com>" | |||
There was a problem hiding this comment.
This would need to have a different author
2 participants
Adds three new ppc64le architecture variant targets, mirroring the existing
x86_64_v2/v3/v4 pattern:
ppc64le_power9-unknown-linux-gnu— compiled with-mcpu=power9 -mtune=power9ppc64le_power10-unknown-linux-gnu— compiled with-mcpu=power10 -mtune=power10ppc64le_power11-unknown-linux-gnu— compiled with-mcpu=power11 -mtune=power11The existing generic
ppc64le-unknown-linux-gnuartifact is untouched.Power9 support is based on Christy Norman's work (clnperez#1).
Power10 uses Debian Bookworm's GCC 12. Power11 uses GCC 14 from Debian Trixie because
GCC 12 does not support
-mcpu=power11.Closes #1215