Reproduction fidelity and limitations#

eb-stack separates deterministic package planning from empirical build evidence. This page states what the checked-in regression suites prove and where an operator must rely on residuals or a target campaign instead of a compatibility assumption.

Evidence boundaries#

Evidence

What it proves

What it does not prove

Parser and package fixtures

Input facts survive normalization into the package plan and planned SBOM

The selected package compiles

Resolvo profile lock

One dependency selection satisfies the declared constraints, pins, exclusions, and EasyBuild hierarchy

The selected versions are ABI-compatible in a real build

Emitted-recipe regression

The rendered .eb text and bundle layout match a frozen expected result

Upstream source bytes, patches, compilers, tests, or sanity succeed

Campaign state

EasyBuild and declared verification commands ran on one named target

The same recipe builds on a different target or runtime ABI

The claim ladder follows these boundaries: planning may establish resolves; only a target campaign may establish builds and binary-verified. New packages enter this boundary through eb-stack package plan; existing recipes use eb-stack package bump; whole-stack selection uses eb-stack stack solve.

Frozen maintainer-bump matrix#

tests/reproduce_real_prs.rs contains 18 library and CLI regressions over six real foss-2023b to foss-2024a recipe pairs:

  • GROMACS 2024.4;

  • ScaFaCoS 1.0.4;

  • MDTraj 1.10.3;

  • Fiona 1.10.1;

  • PuLP 2.8.0;

  • numba 0.60.0.

The zero-hand-fed cases give package bump only the source recipe, target toolchain generation, and frozen EasyBuild universe. Resolvo selects the hierarchy-compatible dependency identities. ScaFaCoS, MDTraj, Fiona, PuLP, and numba reproduce the frozen target content. GROMACS matches after removing the one pybind11 dependency added by the maintainer; the regression also asserts that eb-stack does not invent that source-absent dependency.

tests/bump_emit.rs separately checks canonical bundle paths, deterministic repeat runs, explicit version/checksum inputs, hierarchy-aware selection, and parity between the library and eb-stack package bump CLI paths.

Run the public regression:

cargo test --locked +  --test reproduce_real_prs +  --test bump_emit +  -- --nocapture

This is a fixed regression matrix, not a success-rate estimate for arbitrary EasyBuild recipes.

Foreign-package and profile evidence#

The conda-forge eOn and Spack QMCPACK fixtures exercise a different boundary:

  • static recipe ingestion with source spans and provenance;

  • conda selectors and Spack conditions, variants, conflicts, and requirements;

  • canonical package plans and CycloneDX 1.5 planned SBOMs;

  • per-profile Resolvo locks;

  • one conventional recipe per independently loadable variant;

  • stack-pin fallback and candidate exclusions;

  • packaging gates against frozen overlays and, when configured, a real robot.

Run those deterministic suites:

cargo test --locked +  --test foreign_ingest +  --test foreign_conditions +  --test profile_emission +  --test eon_foss_2026_1 +  --test qmcpack_foss_2026_1 +  --test eon_packaging +  -- --nocapture

These tests establish parser, solver, and emitter behavior. They do not replace campaign run on the selected EasyBuild target.

Real-tree parser scale#

tests/real_tree_scale.rs walks an unfiltered EasyBuild checkout when EB_EASYCONFIGS points at one. It requires more than 1,000 recipes, asserts at least 90% restricted-DSL coverage, and exercises canonical GROMACS, numba, and nglview bumps against that live candidate universe:

EB_EASYCONFIGS=/path/to/easybuild-easyconfigs/easybuild/easyconfigs +  cargo test --locked --test real_tree_scale -- --nocapture

Files outside the restricted EasyBuild DSL are reported as skipped paths with parse errors; one file does not abort the tree walk. See the parser approach for the supported expression model.

Limits that remain explicit#

  • Static foreign parsing: dynamic Python, Jinja, helper calls, or imported logic that cannot be proven from source becomes a residual. The parser does not execute conda-forge or Spack recipe code.

  • Candidate-universe scope: Resolvo can select only identities present in the supplied robot trees. Pass upstream first and overlays afterward so the reviewed site identity wins.

  • Stack policy: a preferred pin may fall back to another jointly feasible candidate and records that outcome. A locked pin makes incompatibility unsatisfiable. Neither mode claims that the candidate builds.

  • Application-version changes: source URLs and checksums cannot be inferred safely. Supply --source-checksum; patches and source-layout changes remain visible residual work.

  • Maintainer additions: a source recipe cannot reveal a dependency, patch, easyblock change, or configuration rule introduced only in the target recipe. Add that authored package knowledge, re-plan, and retain it in the bundle.

  • Planned SBOM scope: the CycloneDX document describes requested components and selections. It is not an installed-files inventory or binary provenance attestation.

  • Target specificity: a build on one host/container/toolchain ABI does not generalize to another. Keep the target name and routed command in campaign evidence.

Unresolved dependencies and unsatisfiable locked pins fail the planning command. Build-compatible claims that fail configure, compile, link, test, install, or sanity become typed campaign findings. Hermes owns the full repair loop; OMP workers claim findings, repair the evidence-supported layer, record the resolution, and rerun the campaign.

Next steps#