Find shared builds across open pull requests¶
Two open pull requests on the European Environment for Scientific
Software Installations (EESSI) software-layer often build the same
dependency. If both are merged and built, the second build overwrites
the first. eb-stack pr-overlap lists the modules each open pull
request (PR) would still build, which of those modules more than one
PR would build, and, with --merge-order, an order that builds each
shared module once.
Use it before you merge a batch of open software-layer PRs that share a dependency. The command analyzes easyconfig and easystack text and writes report files. It does not run EasyBuild or modify a pull request.
Run the recorded fixture¶
This repository records two software-layer PRs under
tests/pr_overlap_layer. Run this from the repository root:
eb-stack pr-overlap --pr tests/pr_overlap_layer/prs.json \
--easyconfigs tests/pr_overlap_layer \
--installed tests/pr_overlap_layer/installed.txt \
--installed-label none \
--generated-at 2026-10-24T00:00:00Z \
--repo-url https://github.com/EESSI/software-layer \
--eessi-version 2026.06 \
--merge-order \
--out-dir /tmp/pr-overlap-out
The command prints:
prs=2 records=2 shared_modules=91 skipped_easyconfigs=0 out=/tmp/pr-overlap-out
prs is how many pull requests were read. records is how many of
them add at least one entry to an easystack for this version. A pull
request with no such entry stays in prs and stays out of records.
Each record has a to_build list, the build set: the modules that PR
would still build. shared_modules counts modules in more than one
build set. skipped_easyconfigs is how many recipe files the parser
could not parse. out is the directory containing overlap.md,
overlap.toml, one prs/<N>.toml file per record, and, with
--merge-order, merge-order.toml.
Pull request 1585 adds SciPy-bundle/2026.05-gfbf-2026.1. Pull
request 1728 adds PLUMED/2.10.0-foss-2026.1, and that recipe depends
on SciPy-bundle 2026.05. The fixture’s installed.txt is a single
comment line, so nothing counts as already built. The fixture is a
replay: both PRs have since merged, and the titles in the report are
the merge-commit subjects stored in prs.json. Both PRs are run
against an empty installed list, which is why they share 91 modules,
including compilers such as GCC/15.2.0.
Read the report¶
The top of overlap.md from this run is:
<!-- Generated by `eb-stack pr-overlap`. This branch is never merged. -->
# Shared builds between open pull requests (2026.06)
Computed 2026-10-24T00:00:00Z against the modules installed for `none`. 2 pull requests change an easystack for this version.
## Modules more than one pull request would build
| Module | Pull requests |
|---|---|
The line under the title names the installed list (none here). The
table has one row per shared module, 91 rows in this run. SciPy-bundle
is one of them:
| `SciPy-bundle/2026.05-gfbf-2026.1` | [#1585](https://github.com/EESSI/software-layer/pull/1585), [#1728](https://github.com/EESSI/software-layer/pull/1728) |
Read that row as: both pull requests would build this module, so one build has to finish before the other starts. Further down, the section for pull request 1585 gives a build count of 91 modules, and the section for pull request 1728 gives 109. All 91 of 1585’s modules are in 1728’s build set.
--merge-order also writes merge-order.toml. These lines are from
that file. Each step’s builds line, and the modules list under
steps.saves, is one long line of module names and is left out here.
[[steps]]
position = 1
pr = 1585
after = []
reason = "smallest residual build set (91)"
[[steps.saves]]
pr = 1728
[[steps]]
position = 2
pr = 1728
after = [1585]
saves = []
reason = "smallest residual build set (18)"
Merge in position order. after lists the PRs that must merge
first. Pull request 1728 must wait for pull request 1585, because
1585 adds SciPy-bundle to the easystack by name, while 1728 only
reaches it as a dependency of PLUMED. When two or more PRs are free to go
next, the one with the fewest modules left to build goes first, and
reason records that count. The word “residual” in reason means
those modules left to build. In step 1, no earlier PR is required,
and pull request 1585 has 91 modules left to build. The
[[steps.saves]] entry says that building those modules for 1585
removes the same 91 modules from 1728’s later build. In step 2, pull
request 1728 has after = [1585], and 18 modules remain, including
PLUMED.
The same order is the Merge order table at the end of overlap.md.
Merge and build pull request 1585 first, and pull request 1728 second.
prs/<N>.toml is that PR’s build set. overlap.toml is the same
shared-module index as data. Field names are in
output formats. A PR omitted from
the current input has no file: each run rewrites prs/ to contain
only records from that run. The live fetch supplies open PRs; static
--pr files are not checked for open or closed status. How one
easystack line becomes a build set, and how that order is chosen, is in
from one easystack line to a build set.
Run it on a mounted stack¶
You need three things: the EESSI repository mounted under
/cvmfs/software.eessi.io, a clone of EESSI/software-layer as the
working directory, and a logged-in gh. The CPU target in this example is
x86_64/amd/zen4. Source the EESSI init script and load the extend
module before the block below:
source /cvmfs/software.eessi.io/versions/2026.06/init/lmod/bash
module load EESSI-extend
That sets EESSI_SOFTWARE_PATH (the install prefix for the CPU target
you loaded) and EASYBUILD_FILTER_DEPS. Set EB_STACK_CHECKOUT to an
eb-stack checkout. Set EB_EASYCONFIGS and EB_EASYBLOCKS to the
easyconfig tree and the easybuild/easyblocks package for the
EasyBuild version these easystacks name.
scripts/fetch-upstream-trees.sh clones a pinned release, which can
differ from that version.
examples/eessi-pr-overlap/fetch-prs.sh reads every open PR and writes
one JSON file per PR that changes an easystack for that EESSI version.
It pages through the GitHub pulls API 100 at a time with no stop at
500, so a repository with more open PRs than one listing returns is
still read in full. Changed files are read 100 per page through
GitHub’s cap of 3000. A pull request whose file list stops there is
still written, with file_list_truncated set, and the record notes
truncated at GitHub's limit. It fetches each PR head into the clone as
refs/remotes/pr/N and does not check it out. --eessi-version
defaults to 2026.06.
: "${EESSI_SOFTWARE_PATH:?source the EESSI init script and module load EESSI-extend}"
: "${EASYBUILD_FILTER_DEPS:?source the EESSI init script and module load EESSI-extend}"
: "${EB_STACK_CHECKOUT:?set this to an eb-stack checkout}"
: "${EB_EASYCONFIGS:?set this to the easyconfig tree for the EasyBuild version these easystacks name}"
: "${EB_EASYBLOCKS:?set this to the easybuild/easyblocks package for that release}"
"$EB_STACK_CHECKOUT/examples/eessi-pr-overlap/fetch-prs.sh" \
EESSI/software-layer 2026.06 /tmp/pr-overlap-in
find "$EESSI_SOFTWARE_PATH/software" -mindepth 2 -maxdepth 2 \
-type d -printf '%P\n' | sort > /tmp/pr-overlap-installed.txt
shopt -s nullglob
args=()
for f in /tmp/pr-overlap-in/prs/*.json; do
args+=(--pr "$f")
done
for d in /tmp/pr-overlap-in/commit-trees/*/; do
args+=(--commit-tree "$(basename "$d")=$d")
done
for d in /tmp/pr-overlap-in/easyblocks/*/; do
args+=(--easyblocks-commit "$(basename "$d")=$d")
done
shopt -u nullglob
eb-stack pr-overlap "${args[@]}" \
--easyconfigs "$EB_EASYCONFIGS" \
--easyblocks "$EB_EASYBLOCKS" \
--installed /tmp/pr-overlap-installed.txt \
--installed-modulerc "$EESSI_SOFTWARE_PATH/modules/all" \
--filter-deps "$EASYBUILD_FILTER_DEPS" \
--installed-label x86_64/amd/zen4 \
--eessi-version 2026.06 \
--exclude-path /accel/ \
--exclude-path /additional_arch/ \
--merge-order \
--out-dir /tmp/pr-overlap-out
On EESSI/software-layer on 2026-10-31, fetch-prs.sh printed:
open pull requests: 52
pull requests with 2026.06 easystack changes: 26
The first line counts every open PR the API returned. The second line
counts the files in prs/, which are the --pr inputs. A PR missing
from prs/ changed no 2026.06 easystack.
nullglob keeps an empty commit-trees/ or easyblocks/ from being
passed as a literal path. You get the same kinds of report files as in
the recorded run. The current PRs and the installed prefix set the
counts and module names. Installed modules drop out of each build set.
shared_modules then counts the uninstalled modules in two or more
build sets.
Flag |
What to pass |
|---|---|
|
The easyconfigs tree for the EasyBuild version the easystack names |
|
The easyconfigs at one |
|
That EasyBuild release’s |
|
The easyblock files one |
|
The prefix’s |
|
Dependencies EasyBuild filters. Defaults to |
See what the bot builds on each architecture¶
The EESSI bot builds more than the lines a PR adds. The bot reads each changed easystack at the PR head. It then builds every entry not yet installed on that architecture, plus the missing dependencies of those entries. An entry can sit in the shared easystack for months and still be missing on one architecture. Every later PR that touches the file makes that bot try it again. Software-layer PR 1775 did not add GDRCopy. The riscv64 bot still failed on it.
Pass --arch-installed ARCH=FILE once per architecture to get that
build set. The file is the same installed-module list as
--installed, and the flag replaces --installed. --bot-build-set
gives the same whole-file closure for a single --installed list.
Without either flag the command closes only the added lines, as in the
recorded run above.
This fixture has the shape of PR 1775. The shared easystack already lists GDRCopy. PR 1775 adds nvtop, and PR 1800 adds likwid. x8664has GDRCopy installed and riscv64 does not.
eb-stack pr-overlap \
--pr tests/pr_overlap_arch/prs.json \
--easyconfigs tests/pr_overlap_arch/release \
--arch-installed x86_64=tests/pr_overlap_arch/x86_64.txt \
--arch-installed riscv64=tests/pr_overlap_arch/riscv64.txt \
--generated-at 2026-10-30T00:00:00Z \
--repo-url https://github.com/EESSI/software-layer \
--merge-order \
--out-dir /tmp/pr-overlap-arch
The command prints:
prs=2 records=2 shared_modules=2 skipped_easyconfigs=0 out=/tmp/pr-overlap-arch
overlap.md now has one section per architecture. On x8664the
shared table says None. and each PR builds only its own module. Here is
the riscv64 section, trimmed:
| `GDRCopy/2.5.2-GCCcore-15.2.0` | [#1775](https://github.com/EESSI/software-layer/pull/1775), [#1800](https://github.com/EESSI/software-layer/pull/1800) |
| `Lib/1.0-GCCcore-15.2.0` | [#1775](https://github.com/EESSI/software-layer/pull/1775), [#1800](https://github.com/EESSI/software-layer/pull/1800) |
Already in the changed easystack and not installed: `GDRCopy/2.5.2-GCCcore-15.2.0`.
The “Already in the changed easystack” line means the PR did not add
GDRCopy, and the riscv64 bot will still build it. Lib comes along
because it is a dependency of GDRCopy. If that architecture cannot
build the module at all, fix the easystack or mark it unsupported
before you merge (see
refuse a plan an architecture cannot build).
The Merge order table is repeated per architecture. On riscv64, PR 1775
goes first and saves GDRCopy and Lib for PR 1800. In prs/1775.toml,
entries is still only the nvtop line, and GDRCopy is listed under
carried. The per-architecture fields are in
output formats.
Wire it into CI¶
The draft workflow is
examples/eessi-pr-overlap/pr_dependency_overlap.yml. It is not
installed in the EESSI software-layer repository from this checkout.
On that repository:
Exempt
gh_action_pr_dependency_overlapfrom the branch push restriction, the same waygh_action_update_software_layer_commit_shais exempt, sogithub-actions[bot]can push to that one branch.mainstays protected.Copy
pr_dependency_overlap.ymlto.github/workflows/andfetch-prs.shto.github/workflows/scripts/fetch_pr_easystacks.sh.Set
EB_STACK_VERSIONandEB_STACK_SHA256in the workflow to a release that carriespr-overlap.
The workflow runs on a schedule and on workflow_dispatch, from
main. It mounts the stack, writes overlap.md, overlap.toml, and
one prs/<N>.toml per recorded PR, then pushes that report to
gh_action_pr_dependency_overlap without force. It opens no pull
request and posts no comment. The publish job skips the commit when only
the generated_at = ~ and ~Computed ~ lines change. A timestamp-only change is not committed. A title change is. ~--impact and --merged exit 2 and name the number
when it is not in the input. Exclusion wins: a path that matches both
--include-path and --exclude-path is skipped.
Easyconfigs pull requests¶
For open PRs on easybuild-easyconfigs, pass --kind easyconfigs. You
do not need --installed: the build set is then the modules the base
tree does not already have. The recorded open set is
tests/pr_overlap_easyconfigs_open/.
eb-stack pr-overlap \
--kind easyconfigs \
--pr tests/pr_overlap_easyconfigs_open/prs.json \
--easyconfigs tests/pr_overlap_easyconfigs_open/tree \
--generated-at 2026-10-24T00:00:00Z \
--repo-url https://github.com/easybuilders/easybuild-easyconfigs \
--out-dir /tmp/ec-open-out
The command prints:
prs=4 records=4 shared_modules=1 skipped_easyconfigs=0 out=/tmp/ec-open-out
22823 and 24575 share Mathematica/14.2.1. That run, including the
live fetch, is in
find open easybuild-easyconfigs pull requests that add the same software.