Run an annual toolchain-generation bump#

The annual workflow applies the canonical bump bundle to each maintained recipe, then drives every bundle through the same target-routed build campaign. Dependency version selection is mechanical; build failures remain typed repair work.

Prepare the generation inputs#

Record:

  • the source and target toolchain generations;

  • upstream EasyBuild robot roots and site overlays;

  • the stack-policy TOML for site/distribution preferences;

  • the public target base plus the site target layer;

  • the maintained recipe list and destination easyconfigs repository.

The site layer owns hostnames, scheduler sizing, module setup, durable build paths, container images, and environment. Do not put those values in the public skill.

Generate one canonical bundle per recipe#

eb-stack package bump \
  --source easyconfigs/g/GROMACS/GROMACS-2024.4-foss-2023b.eb \
  --toolchain-name foss \
  --toolchain-version 2024a \
  --easyconfigs vendor/easybuild-easyconfigs/easybuild/easyconfigs \
  --easyconfigs site/easyconfigs \
  --stack-policy stacks/foss-2024a.toml \
  --out-dir work/GROMACS-2024.4-foss-2024a

Repeat mechanically for the maintained set. Each successful bundle contains a manifest, planned SBOM, default profile lock, and recipe. The lock records stack pin fallbacks and exclusions, making generation policy reviewable.

Validate generated recipes#

eb-stack recipe format work/*/easyconfigs/*/*/*.eb
eb-stack recipe lint work/*/easyconfigs/*/*/*.eb

eb-stack recipe check \
  --recipe work/GROMACS-2024.4-foss-2024a/easyconfigs/g/GROMACS/GROMACS-2024.4-foss-2024a.eb \
  --easyconfigs vendor/easybuild-easyconfigs/easybuild/easyconfigs \
  --easyconfigs site/easyconfigs \
  --easyconfigs work/GROMACS-2024.4-foss-2024a/easyconfigs

Resolve missing dependencies by adding a real package bundle or authored recipe. Checksums are positional: all sources first, then patches. Fix recipe content; never bypass the packaging gate.

Solve the multi-root stack#

Use stack solve when the campaign also needs a generation-wide lock, build list, or baseline diff:

eb-stack stack solve \
  --easyconfigs vendor/easybuild-easyconfigs/easybuild/easyconfigs \
  --easyconfigs site/easyconfigs \
  --policy policies/annual-foss-2024a.json \
  --baseline-easyconfigs vendor/easybuild-easyconfigs/easybuild/easyconfigs \
  --lock-out work/stack.lock.json \
  --sbom-out work/stack.cdx.json \
  --build-list-out work/build.list \
  --stack-diff-out work/stack.diff.md

The whole-stack policy and package stack-policy solve related but distinct problems: JSON chooses a multi-root stack; TOML controls preferred/locked pins and exclusions for one generated product profile.

Validate target routing#

eb-stack target doctor \
  --config examples/targets/base.toml \
  --config ~/.config/eb-stack/site.toml \
  --target site-builder

The doctor must show healthy transport, executor, runtime, and EasyBuild workload checks before a campaign claim is trusted.

Drive every build campaign#

eb-stack campaign run \
  --bundle work/GROMACS-2024.4-foss-2024a \
  --config examples/targets/base.toml \
  --config ~/.config/eb-stack/site.toml \
  --target site-builder \
  --state work/GROMACS-2024.4-foss-2024a.campaign.json

Hermes owns the retry loop. OMP workers claim findings and repair according to class:

Class

Repair surface

transport / executor / runtime

target or site configuration

checksum

source digest and positional checksum list

patch

patch source or applicability

dependency-missing

manifest, profile, stack policy, or real dependency recipe

configure / compile / link

product configuration, toolchain, patch, or dependency selection

test / install / sanity

recipe build logic and verification

resource / timeout

scheduler sizing or retry policy

Each resolution records action, evidence, and changed paths. Rerun the same campaign state; successful stages supersede stale open findings.

Report claims accurately#

  • resolves means every requested profile has a lock and recipe.

  • builds means every emitted recipe succeeds through EasyBuild on the target.

  • binary-verified means every declared verification command succeeds.

An annual bump is not complete while an affected campaign remains failed or a required verification command is absent. The human operator owns branch/PR publication; the campaign supplies the recipe set and evidence.

The executable procedure is skills/annual-bump/SKILL.md.