Dockerfile - Add ROCm6.4 dockerfile

[Hongtao Zhang (polarG)]

Description
Adds a ROCm 6.4.4 SuperBench Docker image targeting AMD MI300X (gfx942), based on #812, plus fixes discovered while building and verifying the image end-to-end on MI300X hardware.

Major Revision

• Add dockerfile/rocm6.4.x.dockerfile — new ROCm 6.4.4 image (Ubuntu 24.04, Python 3.12, PyTorch 2.7.1, gfx942), base rocm/pytorch:rocm6.4.4_ubuntu24.04_py3.12_pytorch_release_2.7.1.
• OFED upgraded to DOCA-Host 25.10 user-space (doca-ofed-userspace=3.2.3-019000), matching the target host's OFED version; MLNX_OFED standalone tarballs are gone upstream.
• Embedded Docker pinned to client-only 29.6.2 (extracts only the docker client binary, dropping dockerd/containerd/runc/shims to shrink the CVE surface — SuperBench only shells out to the Docker client, never runs its own daemon).
• Add flex to the apt package list (fixes an OpenMPI "developer build"/autogen.pl failure that requires it).
• Set AMDGPU_TARGETS=gfx942 explicitly (previously unset, a regression vs rocm6.3.x.dockerfile) — without a GPU present at docker build time, hipcc silently defaulted to gfx906, which caused incorrect zero-copy reads / gpu-copy-bw:correctness failures on MI300X.
• Bump third_party/rccl-tests submodule to 0039629 (adds a missing <cstring> include required for ROCm 6.4's HIP headers, Add cstring header explictly as it is removed from HIP ROCm/rccl-tests#132).
• Fix third_party/Makefile rccl-tests build: repeat the HIP platform defines (-x hip -D__HIP_PLATFORM_AMD__ -D__HIPCC__) inside the HIPCUFLAGS override, since setting AMDGPU_TARGETS causes the override to replace (not append to) rccl-tests' own HIPCUFLAGS, which otherwise drops those defines and breaks the HIP headers.
• Register rocm6.4 in .github/workflows/build-image.yml.

Minor Revision

• CMake if/else shell fix (missing ; before else).
• rm -rf .git cleanup in the final make postinstall step (image size).
• apt-get clean in the apt/OFED install blocks.

Verification
A full sb run was executed on an 8x MI300X host: 460/462 executor runs passed. The 3 failures (resnet rendezvous port collision, gpu-copy-bw:perf OOM, gpu-copy-bw:correctness) were root-caused; the correctness failure was the AMDGPU_TARGETS issue above, confirmed in-container with the full correctness suite passing 321/321 after rebuilding with gfx942. The other two failures are fixed outside this Dockerfile (runner/config changes) and are not blockers for this PR.

[Copilot]

Pull request overview

Adds a new ROCm 6.4.4 SuperBench container image (targeting MI300X/gfx942) and wires it into the CI image build matrix, plus adjusts third-party build flags to keep rccl-tests compiling correctly when AMDGPU_TARGETS is set.

Changes:

• Add dockerfile/rocm6.4.x.dockerfile to build a ROCm 6.4.4 Ubuntu 24.04 / Python 3.12 image with gfx942 targeting and updated dependency/tooling installs.
• Update third_party/Makefile rccl-tests build to preserve required HIP platform defines when overriding HIPCUFLAGS.
• Register the rocm6.4 image in .github/workflows/build-image.yml.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
third_party/Makefile	Updates rccl-tests build flags to stay compatible with AMDGPU_TARGETS overrides under ROCm 6.4.
dockerfile/rocm6.4.x.dockerfile	New ROCm 6.4.4 image definition targeting MI300X (gfx942) and updated build/install steps.
.github/workflows/build-image.yml	Adds rocm6.4 to the image build matrix for CI.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.02%. Comparing base (67298ae) to head (c7ccd24).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #837   +/-   ##
=======================================
  Coverage   86.02%   86.02%           
=======================================
  Files         103      103           
  Lines        7950     7950           
=======================================
  Hits         6839     6839           
  Misses       1111     1111           

Flag	Coverage Δ
cpu-python3.10-unit-test	70.88% <ø> (ø)
cpu-python3.12-unit-test	70.88% <ø> (ø)
cpu-python3.7-unit-test	70.31% <ø> (ø)
cuda-unit-test	83.95% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (3)

dockerfile/rocm6.4.x.dockerfile:74

• The CMake version parsing only captures major.minor (e.g., "3.24"), so patch versions like 3.24.2 are treated as 3.24 and will incorrectly trigger a full CMake source build. This can significantly slow builds and adds avoidable variability.

# Check if CMake is installed and its version
RUN cmake_version=$(cmake --version 2>/dev/null | grep -oP "(?<=cmake version )(\d+\.\d+)" || echo "0.0") && \
    required_version="3.24.1" && \

dockerfile/rocm6.4.x.dockerfile:142

• This cleanup command removes "/tmp/openmpi-${OPENMPI_VERSION}*", but the OpenMPI source tree is cloned into "/tmp/ompi". As written, the build directory is not deleted and will bloat the final image.

    make -j $(nproc) install && \
    ldconfig && \
    cd / && \
    rm -rf /tmp/openmpi-${OPENMPI_VERSION}*

dockerfile/rocm6.4.x.dockerfile:201

• Running find / during docker build will traverse the entire filesystem and can be very slow (and occasionally problematic on special mount points). Since joblib is a Python package, you can locate its installed parallel.py via Python and patch just that file.

RUN pip install "joblib>=1.4.2" && \
    find / -path '*/joblib/parallel.py' -not -path '*/.git/*' -exec sed -i \
        's/timeout_control_job = next(iter(self\._jobs_set), None)/timeout_control_job = next(iter(set(self._jobs_set)), None)/' {} +

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (4)

dockerfile/rocm6.4.x.dockerfile:179

• Same as above: prefer apt-get (non-interactive) and clean apt metadata in the same layer to avoid unnecessary image bloat and reduce the chance of stale package indices causing build failures.

RUN apt install rocm-cmake -y && \

dockerfile/rocm6.4.x.dockerfile:164

• Using apt install in Docker builds is discouraged (apt is geared toward interactive use) and this step also doesn’t clean apt metadata, which can bloat layers and make builds less reproducible. Prefer apt-get update && apt-get install --no-install-recommends ... with cleanup in the same layer.

This issue also appears on line 179 of the same file.

RUN apt install amd-smi-lib -y && \

setup.py:26

• The guidance in the VersionConflict message still hard-pins setuptools==65.7, but for Python >= 3.8 this code now allows any setuptools>=45. Recommending a hard pin can cause unnecessary downgrades and confusion; consider formatting the message using setuptools_req so it stays consistent with the actual requirement being enforced.

    # setuptools 66+ no longer supports Python 3.7, newer runtimes work with any recent version.
    setuptools_req = 'setuptools>=45, <66' if sys.version_info[:2] < (3, 8) else 'setuptools>=45'
    try:
        pkg_resources.require(['pip>=18', setuptools_req])
    except (pkg_resources.VersionConflict, pkg_resources.DistributionNotFound):

.github/workflows/build-image.yml:75

• This workflow’s docker-build job is still using actions/checkout@v2 (see the Checkout step just below), while docker-merge already uses actions/checkout@v4. checkout@v2 is deprecated and can break as the runner images evolve; bumping to v4 here will keep the workflow consistent and avoid Node runtime deprecation issues.

        - name: rocm6.4
          dockerfile: rocm6.4.x
          tags: superbench/main:rocm6.4
          platforms: linux/amd64
          runner: [self-hosted, linux/amd64, rocm]
          build_args: "NUM_MAKE_JOBS=16"

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (3)

dockerfile/rocm6.4.x.dockerfile:166

• Using apt install in a Docker build is discouraged because apt is not a stable CLI for non-interactive scripts, and this layer also doesn't clean apt lists. Prefer apt-get update && apt-get install --no-install-recommends and remove /var/lib/apt/lists/* afterward.

RUN apt install amd-smi-lib -y && \
    cd /opt/rocm/share/amd_smi && \
    python3 -m pip install .

dockerfile/rocm6.4.x.dockerfile:180

• This step also uses apt install (unstable CLI for scripts) and leaves apt lists behind. Switching to apt-get update && apt-get install --no-install-recommends with /var/lib/apt/lists/* cleanup will make builds more reproducible and keep the image smaller.

RUN apt install rocm-cmake -y && \
    python3 -m pip install --upgrade pip wheel "setuptools>=69.0"

dockerfile/rocm6.4.x.dockerfile:124

• The OFED install step runs apt-get update and installs packages but only calls apt-get clean, which leaves /var/lib/apt/lists/* behind and unnecessarily increases the layer size. Consider removing the apt lists after install in this block.

This issue also appears in the following locations of the same file:

• line 164
• line 179

    apt-get update && \
    apt-get install -y --no-install-recommends doca-ofed-userspace=${DOCA_VERSION}-019000 && \
    apt-get clean ; \
    fi

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (1)

dockerfile/rocm6.4.x.dockerfile:190

• The comment says RCCL is "referenced via RCCL_HOME", but RCCL_HOME is not used anywhere in the repository (only declared as a default in third_party/Makefile). This is misleading for future maintainers; RCCL in this image is actually made available via the built artifacts and LD_LIBRARY_PATH.

ADD third_party third_party
# RCCL is built from source above and referenced via RCCL_HOME.
# perftest_rocm6.patch changes are already upstream in the submodule version.

[gusui-msft]

setuptools 66+ still supports Python 3.7, and this setup.py change is unrelated to this PR anyway, consider moving it out.