Spack on EESSI with EESSI installs as externals¶
Concretize a Spack package on top of installs that already have an
easyconfig beside them. Everything the prefix already has comes in as a
Spack external, and Spack builds the rest. The compiler is that prefix’s
GCCcore, and the CPU target is spack arch -t on the machine running the
command.
The prefix in this checkout is CMake 3.31.3 on GCCcore 13.3.0. The Spack
checkout is the eessi branch of
lorisercole/spack, and EasyBuild
5.4.0 is installed in a venv so the script can parse the easyconfigs.
What the script does¶
scripts/eessi-spack-externals.py reads the easyconfig that every install in the European Environment for Scientific Software Installations (EESSI)
keeps under <install>/easybuild/, walks the runtime dependency closure of the
modules you name, and writes a Spack packages.yaml:
GCCcore becomes the
gccexternal with its compiler paths, and arequire: '@<version>'so Spack compiles with it.Toolchains such as
foss,gompiandGCCare skipped. Bundles are not registered themselves; their Python extensions becomepy-*externals, and a dependency on a bundle becomes a dependency on those.With
--spack-packages(the output ofspack list), names Spack does not package are left out and reported on stderr.
Compat-layer packages such as zlib, bzip2, ncurses and openssl are not in EESSI’s
software layer, so they come from spack external find in step 4.
Steps¶
The prefix in this repository is tests/eessi_spack_fixtures. It has
GCCcore 13.3.0 and CMake 3.31.3 built with that GCCcore. The script
records compiler paths bin/gcc, bin/g++, and bin/gfortran under
tests/eessi_spack_fixtures/software/GCCcore/13.3.0.
A Python that imports EasyBuild, and Spack. On Debian the
venvmodule is thepython3-venvpackage.sudo apt-get install -y python3-venv gfortran python3 -m venv ~/.venvs/easybuild ~/.venvs/easybuild/bin/pip install easybuild==5.4.0 git clone --depth 1 --branch eessi https://github.com/lorisercole/spack.git /tmp/spack . /tmp/spack/share/spack/setup-env.sh
Give Spack its own user scope. Do not set
SPACK_DISABLE_LOCAL_CONFIG: it also drops the user scope, and Spack then never reads the generated file.export SPACK_USER_CONFIG_PATH=/tmp/spack-cfg SPACK_USER_CACHE_PATH=/tmp/spack-cache mkdir -p $SPACK_USER_CONFIG_PATH
Generate the externals for the modules the build should reuse. The script writes
packages.yamlon stdout and names missing dependencies on stderr. For this fixture the stderr line isnot in the software layer: CMake/3.31.3-GCCcore-13.3.0 -> ncurses 6.5.~/.venvs/easybuild/bin/python scripts/eessi-spack-externals.py \ --software tests/eessi_spack_fixtures \ --target "$(spack arch -t)" \ --spack-packages tests/eessi_spack_fixtures/spack-packages.txt \ CMake/3.31.3-GCCcore-13.3.0 \ > $SPACK_USER_CONFIG_PATH/packages.yaml
The gcc entry has
require: '@13.3.0'andlanguages:='c,c++,fortran'. The cmake entry depends ongcc@13.3.0.Concretize. Both nodes are externals.
spack spec -I cmake
What to check in the spec¶
On this machine spack arch -t prints icelake, and the spec is:
[e] cmake@3.31.3~doc+ncurses+ownlibs~qtgui build_system=generic build_type=Release platform=linux os=ubuntu24.04 target=icelake %c,cxx=gcc@13.3.0
[e] ^gcc@13.3.0+binutils+bootstrap+futex~graphite+libsanitizer~mold~nvptx~piclibs~profiled~strip build_system=autotools build_type=Release languages:='c,c++,fortran' platform=linux os=ubuntu24.04 target=icelake
Check |
Expected |
|---|---|
compiler on the root |
|
target on the root |
the value of |
|
|
A root on a different gcc version means Spack did not read the generated
file: check SPACK_USER_CONFIG_PATH. An error that an external “there is
no such external spec” names a dependency the script did not emit; rerun
step 3 with --spack-packages.