Reproduction fidelity and limitations

eb-stack separates deterministic package planning from empirical build evidence. This page states what the checked-in regression suites prove and where an operator must rely on residuals or a target campaign instead of a compatibility assumption.

Evidence boundaries

Evidence

What it proves

What it does not prove

Parser and package fixtures

Input facts survive normalization into the package plan and planned SBOM

The selected package compiles

Resolvo profile lock

One dependency selection satisfies the declared constraints, pins, exclusions, and EasyBuild hierarchy

The selected versions are ABI-compatible in a real build

Emitted-recipe regression

The rendered .eb text and bundle layout match a frozen expected result

Upstream source bytes, patches, compilers, tests, or sanity succeed

Campaign state

EasyBuild and declared verification commands ran on one named target

The same recipe builds on a different target or runtime ABI

The claim ladder follows these boundaries: planning may establish resolves; only a target campaign may establish builds and binary-verified. New packages enter this boundary through eb-stack package plan; existing recipes use eb-stack package bump; whole-stack selection uses eb-stack stack solve.

Frozen maintainer-bump matrix

tests/reproduce_real_prs.rs contains 18 library and CLI regressions over six real foss-2023b to foss-2024a recipe pairs:

  • GROMACS 2024.4;

  • ScaFaCoS 1.0.4;

  • MDTraj 1.10.3;

  • Fiona 1.10.1;

  • PuLP 2.8.0;

  • numba 0.60.0.

The zero-hand-fed cases give package bump only the source recipe, target toolchain generation, and frozen EasyBuild universe. Resolvo selects the hierarchy-compatible dependency identities. ScaFaCoS, MDTraj, Fiona, PuLP, and numba reproduce the frozen target content. GROMACS matches after removing the one pybind11 dependency added by the maintainer; the regression also asserts that eb-stack does not invent that source-absent dependency.

tests/bump_emit.rs separately checks canonical bundle paths, deterministic repeat runs, explicit version/checksum inputs, hierarchy-aware selection, and parity between the library and eb-stack package bump CLI paths.

Run the public regression:

cargo test --locked \
  --test reproduce_real_prs \
  --test bump_emit \
  -- --nocapture

This is a fixed regression matrix, not a success-rate estimate for arbitrary EasyBuild recipes.

Foreign-package and profile evidence

The conda-forge eOn and Spack QMCPACK fixtures exercise a different boundary:

  • static recipe ingestion with source spans and provenance;

  • conda selectors and Spack conditions, variants, conflicts, and requirements;

  • canonical package plans and CycloneDX 1.5 planned SBOMs;

  • per-profile Resolvo locks;

  • one conventional recipe per independently loadable variant;

  • stack-pin fallback and candidate exclusions;

  • packaging gates against frozen overlays and, when configured, a real robot.

Run those deterministic suites:

cargo test --locked \
  --test foreign_ingest \
  --test foreign_conditions \
  --test profile_emission \
  --test eon_core_rgpot \
  --test eon_foss_2026_1 \
  --test qmcpack_foss_2026_1 \
  --test eon_packaging \
  -- --nocapture

These tests establish parser, solver, and emitter behavior. They do not replace campaign run on the selected EasyBuild target.

Cross-source LAMMPS control

LAMMPS is a cross-source control for the generic foreign-package path. The conda-forge and Spack fixtures describe the same upstream product through different recipe languages. The conda fixture exercises multiple sources, feedstock patches, selectors, and build/output dependency roles; the Spack fixture exercises version URL generation, resources, variants, conflicts, and conditional dependencies. Static Spack patch directives use the same artifact model as conda patches, including checksum and applicability; version-excluded patches do not leak into the selected build manifest.

Both inputs pass through the same parser-neutral package plan, selector materialization, Resolvo selection, CycloneDX emission, closure expansion, and EasyBuild renderer used by other packages. Production parser code does not dispatch on the LAMMPS package name. examples/packages/lammps.toml is public, authored EasyBuild product policy: it records easyblock parameters, package features, verification commands, and reviewed patches in data rather than embedding those decisions in a parser.

The generated conda closure includes source-only dependencies such as mlip and n2p2 when their Linux selectors apply. Feedstock patches survive package policy through patches_mode = "merge", and same-filename policy entries may replace their metadata without duplicating an artifact. Recipe checks over the root and generated companions establish the resolves rung. They deliberately make no builds or binary-verified claim without a target campaign.

Run the cross-source parser and profile regressions:

cargo test --locked \
  --test foreign_ingest \
  --test foreign_conditions \
  --test package_config \
  --test profile_emission \
  -- --nocapture

Real-tree parser scale

tests/real_tree_scale.rs walks an unfiltered EasyBuild checkout when EB_EASYCONFIGS points at one. It requires more than 1,000 recipes, asserts at least 90% restricted-DSL coverage, and exercises canonical GROMACS, numba, and nglview bumps against that live candidate universe:

EB_EASYCONFIGS=/path/to/easybuild-easyconfigs/easybuild/easyconfigs \
  cargo test --locked --test real_tree_scale -- --nocapture

Files outside the restricted EasyBuild DSL are reported as skipped paths with parse errors; one file does not abort the tree walk. See the parser approach for the supported expression model.

Scoring a bump against a merged PR

The reproduction grind scores an emitted bump against the file an EasyBuild maintainer actually merged. eb_stack::score_reproduction places each file on one of three rungs:

Score

Means

EXACT

Byte-identical to the merged file.

SEMANTIC

Differs only in ways that cannot change the build.

MATERIAL

A real difference in what would be built.

Two rungs used on the scoreboard have no place in this function, because they are decided before any comparison: ERROR (no file was produced) and excluded (the file was never a scoreable bump, such as a toolchain meta-recipe or a backfill). An exclusion is recorded with its reason rather than dropped from the denominator.

EXACT stays raw equality. Only the SEMANTIC against MATERIAL line is drawn on normalized text, by eb_stack::normalize_for_scoring, which removes comment-only lines, comment tails, trailing whitespace, and runs of blank lines. String content survives verbatim: a triple-quoted description carries its blank lines, its indentation, and any # it contains into the built module, so normalizing inside one would call two different module descriptions equivalent.

The distinction is not academic. Scoring PR #25009 (LLVM v21.1.8, merge 3e86bdac) found three divergences in one file: a stale commented-out toolchainopts entry the bump preserved and the merge deleted, one blank-line difference, and a patch set the merge narrowed. A raw text diff reports all three as one MATERIAL verdict, which flattens exactly the distinction the ladder exists to make; the first two cannot change what gets built, and the third builds different sources.

eb_stack::compare_reproduction returns the score together with the raw line diff, never the normalized one. A forgiving verdict is only auditable if the reader can see the difference it forgave.

Normalization that changed a recipe’s meaning would not fail loudly. It would instead make two files that build different things score SEMANTIC. So tests/repro_scoring.rs asserts the property directly: normalizing a recipe leaves the resolved easyconfig identical, over the checked-in fixtures always and over the robot tree when one is present.

EB_EASYCONFIGS=/path/to/easybuild-easyconfigs/easybuild/easyconfigs \
  EB_REQUIRE_EASYCONFIGS=1 \
  cargo test --locked --test repro_scoring -- --nocapture

Limits that remain explicit

  • Static foreign parsing: dynamic Python, Jinja, helper calls, or imported logic that cannot be proven from source becomes a residual. The parser does not execute conda-forge or Spack recipe code. Informational parser notes do not become residuals merely because their text describes dynamic source logic; residual classification is typed at the parser boundary.

  • Candidate-universe scope: Resolvo can select only identities present in the supplied robot trees. Pass upstream first and overlays afterward so the reviewed site identity wins.

  • Stack policy: a preferred pin may fall back to another jointly feasible candidate and records that outcome. A locked pin makes incompatibility unsatisfiable. Neither mode claims that the candidate builds.

  • Application-version changes: source URLs and checksums cannot be inferred safely. Supply --source-checksum; patches and source-layout changes remain visible residual work.

  • Maintainer additions: a source recipe cannot reveal a dependency, patch, easyblock change, or configuration rule introduced only in the target recipe. Add that authored package knowledge, re-plan, and retain it in the bundle.

  • Planned SBOM scope: the CycloneDX document describes requested components and selections. It is not an installed-files inventory or binary provenance attestation.

  • Target specificity: a build on one host/container/toolchain ABI does not generalize to another. Keep the target name and routed command in campaign evidence.

Unresolved dependencies and unsatisfiable locked pins fail the planning command. Build-compatible claims that fail configure, compile, link, test, install, or sanity become typed campaign findings. Hermes owns the full repair loop; OMP workers claim findings, repair the evidence-supported layer, record the resolution, and rerun the campaign.

Next steps