Rebuild the tensorflow base image on python:3.12 - #49
Merged
Conversation
* Install cmake via apt and uninstall the pip package. Its shims in /usr/local/bin shadow /usr/bin/cmake, and they fail with ModuleNotFoundError when a build calls them from inside a pip build isolation environment -- which is why `pip install cellplm` could not build louvain. * Add a test checking that cmake, cpack and ctest resolve to /usr/bin * Bump the version to 1.2.0
nvcr.io stopped publishing tensorflow images after 25.02, which left this image stuck on tensorflow 2.17, numpy 1.26 and scanpy 1.10. * Build on the same python:3.12 base as base_python, and take tensorflow from the and-cuda extra so the CUDA runtimes are pinned to whatever the wheel was built against * Put those CUDA libraries on the linker path with ldconfig. Tensorflow does not look under site-packages/nvidia/*/lib by itself, so without this it finds no GPU and quietly trains on the CPU. * Bump tensorflow 2.17 -> 2.21, numpy 1.26 -> 2.x, scanpy 1.10 -> 1.12 * Check the CUDA libraries are on the linker path in the test -- there is no GPU in CI, but the linker can be asked whether it would find them Verified on a Tesla T4 on the de.NBI cluster: tensorflow finds the GPU, matmul and a keras training step run on it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR rebuilds
base_tensorflow_nvidiaonpython:3.12withtensorflow[and-cuda].nvcr.io stopped publishing TensorFlow images after 25.02,
so the image was pinned to a dead line: TensorFlow 2.17, NumPy 1.26 and Scanpy 1.10, i.e.
dragging pins backwards relative to
base_python. Since the point of the nvcr image wasNVIDIA's optimised build, and that is gone, there is not much reason to stay on it.
Changes:
python:3.12base asbase_python, so this image is now that imageplus one pip extra. Taking CUDA from the
and-cudaextra pins the runtimes to whateverthe TensorFlow wheel was built against, so the two cannot drift apart -- which is the
failure mode that made
senkin_tmp_trainsilently train on the CPU in task_predict_modality.ldconfig, see below.test_tensorflow.pythat those libraries are findable.The image is 12.9 GB.
The linker bit
TensorFlow does not look under
site-packages/nvidia/*/libby itself, so the first GPU runreported
Cannot dlopen some GPU librariesand found no GPU -- the exact silentCPU fallback this PR is supposed to prevent. Driver injection was not the problem;
libcuda.so.1dlopens fine in a plain container. Tested on a T4:python:3.12+tensorflow[and-cuda], as first writtentensorflow/tensorflow:2.21.0-gpuWorth noting that the official TensorFlow image does not work here either, so it is not the
easy way out it looks like.
I used
ldconfigrather thanLD_LIBRARY_PATHso a component cannot clobber it, with atest -sguard so the build fails loudly instead of quietly losing the GPU. There is no GPUin CI, but the linker can be asked whether it would find the libraries, which is enough to
catch a regression.
Verification
On a Tesla T4 (driver 565.57.01) on the de.NBI cluster:
Driver 12.7 against runtime 12.9 is fine via CUDA minor version compatibility, so no driver
upgrade needed on the cluster.
Notes
stardistin task_ist_preprocessing is the only component tracking the moving:1tag,so it is the one that will pick up TensorFlow 2.21 and wants a re-test.
scape,nn_retraining_with_pseudolabelsandgpcountsall pin:1.0.0and are unaffected.🤖 Generated with Claude Code