Download the Bazel release matching the build architecture

[hsinhoyeh]

Problem

ml_metadata/tools/docker_server/Dockerfile hardcodes the x86_64 Bazel installer, so the server image can only be built on x86_64:

curl ... -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh
./bazel-$BAZEL_VERSION-installer-linux-x86_64.sh
rm -f /bazel/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh

This is one reason gcr.io/tfx-oss-public/ml_metadata_store_server is published for amd64 only, which in turn is the last image blocking Kubeflow Pipelines on ARM clusters — see kubeflow/pipelines#10308 and kubeflow/pipelines#10309, open since December 2023.

Change

Bazel publishes a linux-arm64 release for 7.7.0, but as a plain binary rather than an installer script. This selects the artifact from dpkg --print-architecture and installs it directly to /usr/local/bin:

RUN mkdir /bazel && \
    ARCH=$(dpkg --print-architecture) && \
    curl -H "User-Agent: Mozilla/5.0" -fSsL -o /usr/local/bin/bazel \
      https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-linux-${ARCH} && \
    curl -H "User-Agent: Mozilla/5.0" -fSsL -o /bazel/LICENSE.txt \
      https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE && \
    chmod +x /usr/local/bin/bazel

On x86_64 this is equivalent to the previous invocation — the installer's only effect here was to place the same bazel binary on PATH. The LICENSE.txt fetch is unchanged.

Six lines changed, one file.

Verification

Verified on master (be943b8) on an aarch64 host (NVIDIA GB10, Ubuntu 24.04, Docker 28.5.1):

This change works. With it, bazel-7.7.0-linux-arm64 is fetched and installed correctly, and Bazel starts and runs the build on aarch64. Without it the image cannot be built on ARM at all.

master then needs one more thing to finish an aarch64 build, which is outside the scope of this PR. Abseil's stacktrace.cc emits the ARMv8.3 pointer-authentication instruction xpaclri, which the assembler rejects at the default armv8-a baseline:

external/abseil-cpp/absl/debugging/stacktrace.cc [for tool] failed
/tmp/ccRV18O3.s:166: Error: selected processor does not support `xpaclri'

Adding --copt=-march=armv8.3-a --host_copt=-march=armv8.3-a to the bazel build invocation resolves it, and master then builds clean on arm64 with zero errors, producing a working server:

$ docker image inspect ... --format '{{.Architecture}}'
arm64                                    (131 MB)

$ docker run --entrypoint /bin/metadata_store_server ... --help
metadata_store_server: Warning: SetUsageMessage() never called
  Flags from external/com_github_gflags_gflags/src/gflags.cc: ...

Two details worth recording for whoever picks that up:

• --host_copt is required in addition to --copt. The failing target is compiled [for tool], i.e. in the exec configuration, which --copt does not affect. With --copt alone the error persists unchanged.
• -march=armv8-a+pauth would be preferable to bumping the whole baseline to armv8.3-a, since xpaclri is a HINT-space instruction that is a no-op on older cores, and raising the baseline means the binary requires ARMv8.3 hardware (Graviton2, for example, is ARMv8.2). But GCC 9 on the ubuntu:20.04 builder rejects it:

  cc1: error: invalid feature modifier 'pauth' in '-march=armv8-a+pauth'
  

  So a newer builder base image would be the cleaner long-term fix.

I have deliberately not included the -march change here: it needs to be conditional on the target CPU (it is invalid on x86_64), and choosing between raising the baseline and bumping the builder image is a call for maintainers. Happy to send it as a follow-up in whichever form you prefer.

Earlier I also verified the same Dockerfile change against the v1.14.0 tag (Bazel 5.3.0, which likewise publishes linux-arm64), where the full build additionally required making the vendored PostgreSQL pg_config.h architecture-aware — see the note below. Deployed into a Kubeflow Pipelines 2.16.1 install on an arm64 kind cluster, metadata-grpc-deployment reaches 1/1 Running, and ml-pipeline-persistenceagent / ml-pipeline-scheduledworkflow — previously crash-looping against the unavailable metadata service — recover with it.

Note on the remaining aarch64 blocker

On v1.14.0 this Dockerfile change alone was not sufficient — the vendored PostgreSQL build also had to be made architecture-aware. ml_metadata/postgresql.BUILD generates pg_config.h from a fixed list of lines captured from a configure run on x86_64 (PG_VERSION_STR still records "PostgreSQL 12.1 on x86_64-apple-darwin19.2.0"), asserting x86-only CPU capabilities unconditionally. With HAVE__GET_CPUID defined, src/port/pg_bitutils.c includes <cpuid.h>, which only exists on x86:

external/postgresql/src/port/pg_bitutils.c:16:10: fatal error: cpuid.h: No such file or directory

On master that macro is already /* #undef HAVE__GET_CPUID */, and my master build above confirms this is no longer an issue there: #define HAVE_X86_64_POPCNTQ 1 remains unconditional, but it does not break the aarch64 build on its own. So no pg_config.h change is needed for master — I am recording it only because it is still present on release branches such as v1.14.0, and because the underlying pattern (a configure snapshot hardcoded for one CPU, with no select() on architecture) may resurface.

Scoping this PR to the Dockerfile alone keeps it reviewable and independently useful.

[hsinhoyeh]

Updated the description with verification results — the earlier "could not complete a master build" caveat no longer applies.

Summary of what I confirmed on master (be943b8) on aarch64:

1. This change works. bazel-7.7.0-linux-arm64 installs and runs correctly; without it the image cannot be built on ARM at all.
2. The build then stops in Abseil, not in MLMD: stacktrace.cc emits xpaclri (ARMv8.3 pointer authentication), which the assembler rejects at the default armv8-a baseline.
3. Adding --copt=-march=armv8.3-a --host_copt=-march=armv8.3-a clears it, and master then builds clean on arm64 with zero errors, producing a working metadata_store_server (131 MB, arch=arm64, starts and prints its flags).

Two things I'd flag for whoever takes the -march piece:

• --host_copt is genuinely required alongside --copt — the failing target builds [for tool] in the exec configuration, so --copt alone leaves the error unchanged. That cost me a build cycle to spot.
• -march=armv8-a+pauth would be the better fix than raising the baseline to armv8.3-a, since xpaclri is HINT-space and a no-op on older cores, whereas armv8.3-a makes the binary require ARMv8.3 hardware. But GCC 9 on the ubuntu:20.04 builder rejects +pauth outright, so that route needs a newer builder image.

I left the -march change out of this PR deliberately — it must be conditional on target CPU (invalid on x86_64), and the baseline-vs-builder-image tradeoff is your call. Glad to send it as a follow-up in whichever form you prefer.

[hsinhoyeh]

Sent the -march follow-up as #248, stacked on this PR. Together they make the server image build on linux/arm64.

It is gated on dpkg --print-architecture so x86_64 is unaffected, and the body explains why --host_copt is needed alongside --copt (the failing target builds in the exec configuration) and why -march=armv8-a+pauth would be preferable if the builder image were newer than ubuntu:20.04.