Record the verified H200 run; report CPU affinity; v0.6.1 - #45
Merged
Conversation
The first real H200 run came back 7/7 green with a clean crossover — and
with SLURM_CPUS_PER_TASK=6, os.cpu_count()==192, and OMP_NUM_THREADS
unset. Those three together mean the CPU legs of the comparison may have
been oversubscribed, which makes them slower than a correctly configured
run and every GPU speedup correspondingly flattering.
The number that decides it is the AFFINITY MASK, which the notebook was
not reporting. Slurm can constrain a job two ways:
- cgroup CPU quota: you get N cores' worth of time, but the mask still
shows every core on the node. OpenMP sees them all, spawns a thread
each, and thrashes them across your quota.
- cpuset: the mask itself shrinks, OpenMP sees only your allocation,
and an unset OMP_NUM_THREADS is harmless.
Same request, same cpu_count, opposite consequences — so printing
cpu_count and a generic "unset" note, as before, could not distinguish
them. The notebook now prints the mask alongside the node total and the
Slurm request, records the mask in the JSON, and either confirms the
allocation is respected or says plainly that the CPU timings are not
trustworthy and gives the one-line fix.
This matters beyond accuracy: the workshop's CPU-vs-GPU session teaches
that GPUs win for large jobs. A crossover measured against a hobbled CPU
is exactly the claim a room of computational chemists will challenge.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first real H200 allocation came back 7/7 green with a clean CPU->GPU crossover, so the pairing that worked is now written down where the next build can compare against it: driver 580.126.20, compute capability 9.0, partition gpu, cuda12x wheels. Notably CuPy reports driver API 13000 against runtime 12090 — exactly the backward-compatible combination that made cuda12x the right call over cuda13x, which would have hard-failed on the 570-series driver the hardware notes originally listed. The diagnostic notebook now reports the CPU AFFINITY MASK, not just the core count. Slurm can grant 6 cores two ways: a cgroup quota (the mask still shows all 192, so OpenMP spawns 192 threads and thrashes them) or a cpuset (the mask shrinks and an unset OMP_NUM_THREADS is harmless). Same request, same cpu_count, opposite consequences — so the number the notebook was printing could not distinguish them. It now states plainly when CPU timings cannot be trusted, and gives the one-line fix. On this run the mask was 6, so the measured crossover stands. It is recorded with its CPU allocation, because a speedup without its denominator is the first thing an audience challenges — and the node has ~12 cores per GPU, so 6 is less than a proportional share. Version 0.6.1: this release is the reorganization-energy work merged in #44 plus these docs. Patch rather than minor — nothing here changes an API or removes a capability. Full suite: 2153 passed, 23 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first real H200 allocation came back 7/7 green with a clean CPU→GPU crossover. This records the pairing that worked, closes a measurement trap the run exposed, and bumps to v0.6.1.
Verified on NCShare
gpu/compute-gpu-02cuda12xwheelsCuPy reports driver API 13000 against runtime 12090 — precisely the backward-compatible combination that made
cuda12xthe right call.cuda13xwould have hard-failed on the 570-series driver the hardware notes originally listed, so the conservative choice paid off.Measured crossover (1 GPU vs 6 CPU cores):
GPU wall time barely moves across the first four — fixed launch and transfer overhead, visible rather than asserted. Only at cc-pVTZ does the arithmetic grow enough for the device to matter.
The trap this run exposed
os.cpu_count()reported 192 while Slurm had granted 6, withOMP_NUM_THREADSunset. Whether that matters depends entirely on how Slurm constrains the job:OMP_NUM_THREADSis harmless.Same request, same
cpu_count, opposite consequences — so the number the notebook was printing could not distinguish them. It now reports the affinity mask, records it in the JSON, and either confirms the allocation is respected or states plainly that CPU timings are not trustworthy, with the fix.On this run the mask was 6, so the numbers above stand. The check exists so that is a fact rather than an assumption.
Why the allocation is quoted with the speedup
These are 1 GPU vs 6 cores; the node has ~12 physical cores per GPU, so a proportional-share comparison uses ~12 and would show a smaller factor. The notebook now prints the core count in its summary and advises running at both — the crossover shape is identical, but "why only 6 cores?" is the first question an audience asks, and a number without its denominator invites it.
Version
0.6.1 — patch. This release is the reorganization-energy work merged in #44 plus these docs; nothing here changes an API or removes a capability. Bumped in
pyproject.toml,__init__.py, and the GPU container pin (a test keeps the three in step).Full suite: 2153 passed, 23 skipped.
🤖 Generated with Claude Code