Skip to content

Rebuild the tensorflow base image on python:3.12 - #49

Merged
rcannood merged 3 commits into
mainfrom
rebuild-tensorflow-base
Aug 13, 2026
Merged

Rebuild the tensorflow base image on python:3.12#49
rcannood merged 3 commits into
mainfrom
rebuild-tensorflow-base

Conversation

@rcannood

Copy link
Copy Markdown
Member

This PR rebuilds base_tensorflow_nvidia on python:3.12 with tensorflow[and-cuda].

nvcr.io stopped publishing TensorFlow images after 25.02,
so the image was pinned to a dead line: TensorFlow 2.17, NumPy 1.26 and Scanpy 1.10, i.e.
dragging pins backwards relative to base_python. Since the point of the nvcr image was
NVIDIA's optimised build, and that is gone, there is not much reason to stay on it.

Changes:

  • Build on the same python:3.12 base as base_python, so this image is now that image
    plus one pip extra. Taking CUDA from the and-cuda extra pins the runtimes to whatever
    the TensorFlow wheel was built against, so the two cannot drift apart -- which is the
    failure mode that made senkin_tmp_train silently train on the CPU in task_predict_modality.
  • Bump TensorFlow 2.17 -> 2.21, NumPy 1.26 -> 2.x, Scanpy 1.10 -> 1.12.
  • Put the CUDA libraries on the linker path with ldconfig, see below.
  • Check in test_tensorflow.py that those libraries are findable.

The image is 12.9 GB.

The linker bit

TensorFlow does not look under site-packages/nvidia/*/lib by itself, so the first GPU run
reported Cannot dlopen some GPU libraries and found no GPU -- the exact silent
CPU fallback this PR is supposed to prevent. Driver injection was not the problem;
libcuda.so.1 dlopens fine in a plain container. Tested on a T4:

image GPU detected
python:3.12 + tensorflow[and-cuda], as first written no
same, with the CUDA libraries on the linker path yes
official tensorflow/tensorflow:2.21.0-gpu no

Worth noting that the official TensorFlow image does not work here either, so it is not the
easy way out it looks like.

I used ldconfig rather than LD_LIBRARY_PATH so a component cannot clobber it, with a
test -s guard so the build fails loudly instead of quietly losing the GPU. There is no GPU
in CI, but the linker can be asked whether it would find the libraries, which is enough to
catch a regression.

Verification

On a Tesla T4 (driver 565.57.01) on the de.NBI cluster:

GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Created device ... Tesla T4, compute capability 7.5, 13762 MB
matmul ran on: .../device:GPU:0
Loaded cuDNN version 92400  |  Compiled cluster using XLA!
keras training OK  ->  GPU TEST PASSED

Driver 12.7 against runtime 12.9 is fine via CUDA minor version compatibility, so no driver
upgrade needed on the cluster.

Notes

  • Stacked on Drop the pip-installed cmake from the pytorch base image #48, so please merge that one first. The diff here is tensorflow-only.
  • stardist in task_ist_preprocessing is the only component tracking the moving :1 tag,
    so it is the one that will pick up TensorFlow 2.21 and wants a re-test. scape,
    nn_retraining_with_pseudolabels and gpcounts all pin :1.0.0 and are unaffected.

🤖 Generated with Claude Code

* Install cmake via apt and uninstall the pip package. Its shims in
  /usr/local/bin shadow /usr/bin/cmake, and they fail with
  ModuleNotFoundError when a build calls them from inside a pip build
  isolation environment -- which is why `pip install cellplm` could not
  build louvain.
* Add a test checking that cmake, cpack and ctest resolve to /usr/bin
* Bump the version to 1.2.0
nvcr.io stopped publishing tensorflow images after 25.02, which left this
image stuck on tensorflow 2.17, numpy 1.26 and scanpy 1.10.

* Build on the same python:3.12 base as base_python, and take tensorflow
  from the and-cuda extra so the CUDA runtimes are pinned to whatever the
  wheel was built against
* Put those CUDA libraries on the linker path with ldconfig. Tensorflow
  does not look under site-packages/nvidia/*/lib by itself, so without
  this it finds no GPU and quietly trains on the CPU.
* Bump tensorflow 2.17 -> 2.21, numpy 1.26 -> 2.x, scanpy 1.10 -> 1.12
* Check the CUDA libraries are on the linker path in the test -- there is
  no GPU in CI, but the linker can be asked whether it would find them

Verified on a Tesla T4 on the de.NBI cluster: tensorflow finds the GPU,
matmul and a keras training step run on it.
Base automatically changed from fix-pytorch-cmake-shim to main August 13, 2026 17:35
@rcannood
rcannood merged commit 443b8ba into main Aug 13, 2026
@rcannood
rcannood deleted the rebuild-tensorflow-base branch August 13, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant