GPU builds in EESSI container are broken with Apptainer 1.5.0

[bedroge]

Apptainer 1.5.0 automatically adds system library locations to $LD_LIBRARY_PATH if host libraries are bind mounted to /.singularity/libs (which is done for the --nv flag, i.e. for GPU builds), see https://github.com/apptainer/apptainer/blob/main/CHANGELOG.md#changed-defaults--behaviours. This breaks our startprefix script, as it will pick up the glibc from the host:

/cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/bin/bash: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/bin/bash)


Apptainer> echo $LD_LIBRARY_PATH
/usr/lib/x86_64-linux-gnu/libfakeroot:/usr/local/lib:/lib/x86_64-linux-gnu:/lib:/.singularity.d/libs

Older versions did not do this, they would only set:

Apptainer> echo $LD_LIBRARY_PATH
/.singularity.d/libs

[bedroge]

Possible workaround:

APPTAINERENV_LD_LIBRARY_PATH=/.singularity.d/libs

This overrides the container's $LD_LIBRARY_PATH, and basically results in the same behavior as with older versions.

[ocaisa]

This could also be "solved" with #114 in a more bullet-proof way

[casparvl]

Just to clarify:

APPTAINERENV_LD_LIBRARY_PATH=/.singularity.d/libs

can be put in the bot's site configuration script.

We can also easily put it into our bot/build.sh script => Discussed in support meeting, we should just do this. The issue only pops up for GPU builds, but we can set this unconditionally.

Note that if you don't use the bot, but still use the EESSI container, you will also run into this issue.