Warn about known per-architecture failures

What it does: stack known-failures reads an easystack as a plan, closes its dependencies over --easyconfigs, and warns about every entry a bot run has already failed on one of the architectures you name. Each warning names the easyconfig, the targets, the symptom, and the remedy the maintainers chose, as easyconfig text.

Why you’d use it: these entries build. They fail on one target, in a test, a test timeout, a fetch, or a sanity check, and the usual first answer in the pull request is a rerun. Software-layer pull request 1670 added MultiQC 1.28. MultiQC depends on Kaleido 0.2.1, and the aarch64 bots could not fetch its wheel: the easyconfig asked for manylinux1_aarch64, and PyPI only has manylinux2014 for aarch64.

eb-stack stack known-failures \
  --arch aarch64 \
  --easystack tests/arch_known_failures_fixtures/easystacks/1670-first/eessi-2025.06-eb-5.4.0-2024a.yml \
  --easyconfigs tests/arch_known_failures_fixtures/easyconfigs

The command prints:

Kaleido-0.2.1-GCCcore-13.3.0.eb on aarch64: known fetch failure: kaleido-0.2.1-py2.py3-none-manylinux1_aarch64.whl does not exist; the aarch64 wheel is manylinux2014 (required by MultiQC-1.28-gfbf-2024a.eb)
  remedy: make the wheel an ARCH conditional: local_manylinux = 'manylinux2014' if ARCH == 'aarch64' else 'manylinux1', with one checksum per wheel
  seen in: software-layer #1670; easybuild-easyconfigs #26981
known=1 remedied=0

The first line is the warning. Kaleido-0.2.1-GCCcore-13.3.0.eb is the easyconfig that fails, aarch64 is where, and fetch failure is the phase. required by says the plan did not name Kaleido; MultiQC’s closure did. remedy: is what to change in the easyconfig. seen in: is where the table entry came from. known=1 counts the warnings. The command exits 0: the warning is for the pull request description, or for a fix before the bot runs.

The table is data/arch-known-failures.toml. Every entry cites the software-layer pull request and comment it came from.

A remedy that is already in place

Pull request 1670 was merged with Kaleido pinned to the easyconfigs commit that has the fix:

eb-stack stack known-failures \
  --arch aarch64 \
  --easystack tests/arch_known_failures_fixtures/easystacks/1670/eessi-2025.06-eb-5.4.0-2024a.yml \
  --easyconfigs tests/arch_known_failures_fixtures/easyconfigs
Kaleido-0.2.1-GCCcore-13.3.0.eb on aarch64: known fetch failure, remedied: from-commit be78248 carries the remedy
known=0 remedied=1

A from-commit or include-easyblocks-from-commit pin listed in the entry’s fixed_by makes the line remedied:. So does an easyconfig in --easyconfigs that already contains the entry’s fixed_marker: the fixed Kaleido easyconfig has manylinux2014, and the fixed waLBerla 7.2 easyconfig has the new checksum of its SVE constexpr patch.

A failure an EESSI hook causes

EESSI’s inject_gpu_property hook moves CUDA from dependencies to builddependencies. EasyBuild 5.4 no longer loads build dependencies for the sanity check, so the GROMACS easyblock thinks it built without CUDA and looks for bin/gmx_d. That failed pull request 1658 on zen5:

eb-stack stack known-failures \
  --arch x86_64/amd/zen5 \
  --easystack tests/arch_known_failures_fixtures/easystacks/1658-first/accel/nvidia/eessi-2026.06-eb-5.4.0-2026.1.yml \
  --easyconfigs tests/arch_known_failures_fixtures/easyconfigs
GROMACS-2026.3-foss-2026.1-CUDA-13.3.0.eb on x86_64/amd/zen5: known sanity check failure: CUDA is only a build dependency after the hook, EasyBuild 5.4 does not load it for the sanity check, and the easyblock then expects bin/gmx_d
  caused by: the EESSI inject_gpu_property hook
  remedy: pin include-easyblocks-from-commit f6f23d4, whose GROMACS easyblock checks 'CUDA' in the dependency names instead of the loaded module
  seen in: software-layer #1658; easybuild-easyblocks #4234
known=1 remedied=0

caused by: names the hook. The entry has requires_dependency = "CUDA", so a GROMACS without CUDA is not warned about, and min_easybuild = "5.4.0", so an easystack named eb-5.3.0 is not either. --easybuild-version overrides the version in the file name. On other entries, in EESSI: names the hook that already applies the remedy, such as pre_single_extension_orjson for the a64fx orjson test.

Failing a pre-bot check on a remaining warning

By default a warning is advice and the command exits 0. Pass --strict to exit 1 while any warning remains, so a pre-bot script or CI job fails on a known failure that has no remedy in place yet:

eb-stack stack known-failures --strict \
  --arch aarch64/a64fx \
  --easystack tests/arch_known_failures_fixtures/easystacks/1829/eessi-2026.06-eb-5.4.1-2026.1.yml \
  --easyconfigs tests/arch_known_failures_fixtures/easyconfigs

The stdout is the same report as without --strict; the error on stderr names each remaining warning by easyconfig and target, here orjson-3.11.8-GCCcore-15.2.0.eb on aarch64/a64fx. A remedied hit never counts, so the merged 1670 easystack above exits 0 under --strict.

Choosing architectures

--arch repeats and takes a target or a family. --arch aarch64 meets every aarch64 target in the table, and the line names the target, as in aarch64/a64fx. No --arch checks every architecture the table names. --installed drops modules that are already installed, because the bot does not rebuild them. --eessi-version overrides the eessi-YYYY.MM in the file name. The haswell rustc segfault entry is for 2025.06 only.

stack batch runs the same check for each batch it writes. The warnings go into pr-body.md and batch.json, and plan.md counts them. --known-failures replaces the table on both commands.

See the command-line reference and refuse a plan an architecture cannot build for entries that cannot build at all.