Warn before a bot job runs out of memory¶
What it does: stack oom reads an easystack as a plan. For each entry
and each EESSI bot build node, it estimates the peak build memory at the
parallel level the bot will use. When that estimate reaches the job’s
memory, it prints a warning with remedies. The remedies are text only.
The command never edits a recipe or a hook, and it always exits 0.
Why you’d use it: a bot job that runs out of memory fails late and
quietly. Software-layer pull request 1772 added
libxc-7.0.0-GCC-15.2.0.eb. The build was killed for lack of memory on
zen4 twice at 8 processes on a 32 GiB job, and again at 5 after a hook
change. The same thing happened to Highway in #1692, to
hipBLASLt and rocBLAS in #1549, and to PETSc in #1608. Run the check
before you open the pull request or ask the bot to build.
eb-stack stack oom \
--easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml
The command prints:
libxc-7.0.0-GCC-15.2.0.eb may run out of memory on x86_64/amd/zen4 (eessi-bot-mc-aws amd-zen4): 8 x 6.4 GiB = 51.2 GiB, the job has 32.0 GiB (software-layer #1772, software-layer-scripts #337)
parallel 8: 16 cores, eb_hooks.py divides libxc by 2
why: cc1 on the generated mgga_x_*.c sources; with GCC 15.2.0 on zen4, 8 and then 5 compilers ran out of a 32 GiB job
remedy: maxparallel = 4 in libxc-7.0.0-GCC-15.2.0.eb, or a libxc limit of 4 for x86_64/amd/zen4 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 6.4 GiB per process is a lower bound, so 4 may still be too many
remedy: build x86_64/amd/zen4 on a node with more memory: eessi-bot-surf amd-zen4 (16 cores, 180.0 GiB, accel nvidia/cc90)
remedy: a libxc divider of 3 left 5 cores and ran out too; a reduction of 8, as asked in the review of software-layer-scripts #337 (issuecomment-6599147327), is 2 cores on the AWS build cluster
oom=1 fits=14 builds=5 nodes=15
How to read it:
The first line names the entry, the target, the bot instance and the node, and the estimate against the job’s memory. The pull request numbers are where the figure was measured.
parallel 8is how the bot gets to 8. The node has 16 cores, and the EESSIpost_ready_hookhalves libxc.whysays what used the memory in the real job.Each
remedyis a separate way out: a lowermaxparallelor hook limit, a node with more memory for the same target, and what the maintainers did or asked for.The count line says one entry and node pair may run out of memory, 14 pairs have a profile and fit, the plan has 5 builds, and 15 nodes were checked. Without
--arch, every node in the bot resource table is checked.
A dependency the easystack does not name¶
With --easyconfigs, the command closes the plan’s dependencies, so a
dependency that is not installed yet is checked too. Pull request 1692
added only eOn. Highway came in as a dependency and failed on
aarch64/generic:
eb-stack stack oom \
--arch aarch64/generic \
--easystack tests/oom_profile_fixtures/easystacks/pr1692-eessi-2026.06-eb-5.4.0-2026.1.yml \
--easyconfigs tests/oom_profile_fixtures/easyconfigs
Highway-1.4.0-GCCcore-15.2.0.eb may run out of memory on aarch64/generic (eessi-bot-mc-aws generic): 16 x 1.9 GiB = 30.4 GiB, the job has 29.8 GiB (software-layer #1692, software-layer-scripts #317) (required by eOn-3.6.0-foss-2026.1.eb)
parallel 16: 16 cores, no eb_hooks.py limit for Highway
why: compiling the hwy/tests sources exhausts virtual memory
remedy: maxparallel = 15 in Highway-1.4.0-GCCcore-15.2.0.eb, or a Highway limit of 15 for aarch64/generic in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 1.9 GiB per process is a lower bound, so 15 may still be too many
remedy: no other aarch64/generic node in the bot resource table has room for it
remedy: software-layer-scripts #317 halves Highway on every target and keeps A64FX at a quarter
oom=1 fits=0 builds=2 nodes=1
(required by eOn-3.6.0-foss-2026.1.eb) says which easystack entry
pulled Highway in. Add --installed with one name/version-toolchain
module per line to skip what the target already has.
When fewer cores do not help¶
Some builds need a fixed amount of memory whatever the parallel level. Pull request 1549 built rocBLAS for ROCm 6.4.1, and hipBLASLt came in as its dependency:
eb-stack stack oom \
--arch x86_64/amd/zen2 \
--easystack tests/oom_profile_fixtures/easystacks/pr1549-eessi-2025.06-eb-5.3.1-2025a.yml \
--easyconfigs tests/oom_profile_fixtures/easyconfigs
hipBLASLt-0.12.1-rocm-compilers-19.0.0-ROCm-6.4.1.eb may run out of memory on x86_64/amd/zen2 (eessi-bot-mc-aws amd-zen2): 90.0 GiB at any parallel level, the job has 27.9 GiB (software-layer-scripts #281, software-layer #1549) (required by rocBLAS-4.4.0-rocm-compilers-19.0.0-ROCm-6.4.1.eb)
parallel 8: 16 cores, eb_hooks.py caps hipBLASLt at 8
why: Tensile kernel generation for gfx942 holds about 90 GiB whatever the parallel level
remedy: a lower parallel level does not help: one process already needs 90.0 GiB
remedy: no other x86_64/amd/zen2 node in the bot resource table has room for it
remedy: the hipBLASLt and rocBLAS jobs ran on the larger -xl build nodes, through a temporary bot config (software-layer-scripts #302)
rocBLAS-4.4.0-rocm-compilers-19.0.0-ROCm-6.4.1.eb may run out of memory on x86_64/amd/zen2 (eessi-bot-mc-aws amd-zen2): 50.0 GiB at any parallel level, the job has 27.9 GiB (software-layer-scripts #302, software-layer #1549)
parallel 8: 16 cores, eb_hooks.py caps rocBLAS at 8
why: Tensile kernel generation for gfx942 needs more than 50-60 GB
remedy: a lower parallel level does not help: one process already needs 50.0 GiB
remedy: no other x86_64/amd/zen2 node in the bot resource table has room for it
remedy: the hipBLASLt and rocBLAS jobs ran on the larger -xl build nodes, through a temporary bot config (software-layer-scripts #302)
oom=2 fits=0 builds=2 nodes=1
The estimate says at any parallel level, so the only remedy is a
bigger node.
Check a hook change¶
The parallel level comes from the hook limits in
data/eessi-bot-resources.toml. Pass --resources to try other limits
or other nodes, or --hooks to read the limits from an eb_hooks.py
(see Read the limits from eb:sub:`hooks.py`_). Pull request 1608
ran out of memory in the PETSc test step on zen5 before
software-layer-scripts #298 halved PETSc. With the current hooks, PETSc
fits:
eb-stack stack oom \
--arch x86_64/amd/zen5 \
--easystack tests/oom_profile_fixtures/easystacks/pr1608-eessi-2025.06-eb-5.4.0-2025b.yml \
--easyconfigs tests/oom_profile_fixtures/easyconfigs
oom=0 fits=1 builds=3 nodes=1
The fixture resources-before-298.toml has the zen5 node and the hooks
as they were before #298:
eb-stack stack oom \
--resources tests/oom_profile_fixtures/resources-before-298.toml \
--easystack tests/oom_profile_fixtures/easystacks/pr1608-eessi-2025.06-eb-5.4.0-2025b.yml \
--easyconfigs tests/oom_profile_fixtures/easyconfigs
PETSc-3.24.0-foss-2025b.eb may run out of memory on x86_64/amd/zen5 (eessi-bot-aws-eu-south amd-zen5): 16 x 2.0 GiB = 32.0 GiB, the job has 31.5 GiB (software-layer #1608, software-layer-scripts #298) (required by slepc4py-3.24.0-foss-2025b.eb)
parallel 16: 16 cores, no eb_hooks.py limit for PETSc
why: the test step runs make -j <parallel> test, and that is what ran out
remedy: maxparallel = 15 in PETSc-3.24.0-foss-2025b.eb, or a PETSc limit of 15 for x86_64/amd/zen5 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 2.0 GiB per process is a lower bound, so 15 may still be too many
remedy: no other x86_64/amd/zen5 node in the bot resource table has room for it
remedy: software-layer-scripts #298 halves PETSc on every target
oom=1 fits=0 builds=3 nodes=1
Where the numbers come from¶
Two data files hold the numbers, and every row cites a real pull request or bot configuration.
data/build-memory.tomlholds the peak build memory per software and version family. An entry may be narrowed to a toolchain family and a target. The estimate isbase_gib + per_process_gib * parallel. Each entry lists the bot jobs it came from, with the parallel level, the job’s memory, and whether it ran out. The table is refused when loading if it does not reproduce every one of those jobs, so a new figure cannot contradict an old observation. A figure markedlower_boundis the least that explains a failure. The real peak may be higher.data/eessi-bot-resources.tomlholds each bot build node: the instance, the node, the target, the cores and the job’s memory. It also holds EasyBuild’s default maximum parallel level of 16 and thePARALLELISM_LIMITStable from the EESSIeb_hooks.py.
The parallel level is the smallest of these: the node’s cores, or
--parallel in their place; EasyBuild’s maximum; the hook limit applied
to the cores; and a top-level maxparallel in the recipe. The hook ops are the ones in
eb_hooks.py. divide_by_factor(parallel, factor) is
max(1, parallel // factor). set_maximum(parallel, n) is
min(parallel, n). A software name uses its limit for the CPU target,
and that key overrides its '*' limit. Only software with no hook entry
gets the target’s own limit, which divides by 4 on A64FX. The hook
applies the op to the cores the job can use, and keeps the result only
when it is lower than the parallel level EasyBuild already chose.
When more than one entry matches a build, the highest estimate wins.
divide in the hook table is an integer in eb_hooks.py. A factor that
is not an integer makes Python’s // return a float: 16 // 2.5 is
6.0, and max keeps that float when it is at least 1. A whole number
written as a float does the same: 16 // 2.0 is 8.0. The command
warns, and the estimate uses the numeric floor. Write the factor as an
integer.
In post_ready_hook the log line is two statements:
msg = "limiting parallelism to %s (was %s, derived parallelism %s) for %s on %s "
msg+ "to avoid out-of-memory failures during building/testing"
The second statement adds nothing. msg + "..." is an expression, and
the result is thrown away. The operator that appends is +=. The line
that prints stops after the CPU target. This is upstream eb_hooks.py.
eb-stack does not patch it. --hooks reports it with its line (see
Read the limits from eb:sub:`hooks.py`_).
A divide factor that is not an integer¶
The maintainers asked for a libxc divider of 2.5 or 3, which is 6 or 5
cores on the 16-core AWS build nodes. 3 is an integer. 2.5 is not, and
eb_hooks.py would store a float. The fixture
resources-divide-2.5.toml is that factor, not the integer 2 the hooks
ship:
eb-stack stack oom \
--arch x86_64/amd/zen4 \
--resources tests/oom_profile_fixtures/resources-divide-2.5.toml \
--easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml
warning: PARALLELISM_LIMITS divide_by_factor for libxc on * uses 2.5, which is not an integer; eb_hooks.py computes max(1, parallel // factor), and 16 // 2.5 is 6.0, a float. Use an integer factor.
libxc-7.0.0-GCC-15.2.0.eb may run out of memory on x86_64/amd/zen4 (eessi-bot-mc-aws amd-zen4): 6 x 6.4 GiB = 38.4 GiB, the job has 32.0 GiB (software-layer #1772, software-layer-scripts #337)
parallel 6: 16 cores, eb_hooks.py divides libxc by 2.5 (16 // 2.5 is 6.0, a float)
why: cc1 on the generated mgga_x_*.c sources; with GCC 15.2.0 on zen4, 8 and then 5 compilers ran out of a 32 GiB job
remedy: maxparallel = 4 in libxc-7.0.0-GCC-15.2.0.eb, or a libxc limit of 4 for x86_64/amd/zen4 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 6.4 GiB per process is a lower bound, so 4 may still be too many
remedy: no other x86_64/amd/zen4 node in the bot resource table has room for it
remedy: a libxc divider of 3 left 5 cores and ran out too; a reduction of 8, as asked in the review of software-layer-scripts #337 (issuecomment-6599147327), is 2 cores on the AWS build cluster
oom=1 fits=0 builds=5 nodes=1
The first warning is the float. The estimate then uses 6, and 6 times 6.4 GiB is 38.4 GiB, more than the 32 GiB zen4 job. The divider of 3 ran out at 5 in bot job 221273 of #1772, so 6 does not fit either. The fixture has only the AWS zen4 node, so no other node has room. The command still exits 0.
Read the limits from ebhooks.py¶
The hook rows in data/eessi-bot-resources.toml are a copy of
PARALLELISM_LIMITS. --hooks reads that dict from an eb_hooks.py
instead, so a hook change can be checked before it is merged. The file
is parsed as Python, and nothing in it runs. The fixture
hooks/divide-2.5.py has the layout of the real file and the libxc
divider of 2.5:
eb-stack stack oom \
--arch x86_64/amd/zen4 \
--hooks tests/oom_profile_fixtures/hooks/divide-2.5.py \
--easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/divide-2.5.py
warning: tests/oom_profile_fixtures/hooks/divide-2.5.py:20: PARALLELISM_LIMITS divide_by_factor for libxc on * uses 2.5, which is not an integer; eb_hooks.py computes max(1, parallel // factor), and 16 // 2.5 is 6.0, a float. Use an integer factor.
libxc-7.0.0-GCC-15.2.0.eb may run out of memory on x86_64/amd/zen4 (eessi-bot-mc-aws amd-zen4): 6 x 6.4 GiB = 38.4 GiB, the job has 32.0 GiB (software-layer #1772, software-layer-scripts #337)
parallel 6: 16 cores, eb_hooks.py divides libxc by 2.5 (16 // 2.5 is 6.0, a float)
why: cc1 on the generated mgga_x_*.c sources; with GCC 15.2.0 on zen4, 8 and then 5 compilers ran out of a 32 GiB job
remedy: maxparallel = 4 in libxc-7.0.0-GCC-15.2.0.eb, or a libxc limit of 4 for x86_64/amd/zen4 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 6.4 GiB per process is a lower bound, so 4 may still be too many
remedy: build x86_64/amd/zen4 on a node with more memory: eessi-bot-surf amd-zen4 (16 cores, 180.0 GiB, accel nvidia/cc90)
remedy: a libxc divider of 3 left 5 cores and ran out too; a reduction of 8, as asked in the review of software-layer-scripts #337 (issuecomment-6599147327), is 2 cores on the AWS build cluster
oom=1 fits=1 builds=5 nodes=2
The first line says how many rows were read, and from which file. The
first warning names the line of the (divide_by_factor, 2.5) tuple.
The nodes still come from the bot resource table, so both zen4 nodes are
checked, and the 180 GiB node has room for 6. With every factor an
integer there is no warning: line:
eb-stack stack oom --hooks tests/oom_profile_fixtures/hooks/integers.py
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/integers.py
The libxc and A64FX rows of integers.py are the ones in the bot
resource table, so --node (see
A memory limit instead of a core fraction) prints the same
parallel levels with and without it. For the real hooks, pass the file from a software-layer-scripts
checkout, --hooks software-layer-scripts/eb_hooks.py.
A key is a string or a module-level string constant such as
CPU_TARGET_A64FX. A key the dict repeats keeps its last value, as in
Python. An op other than divide_by_factor or set_maximum, a \*\*
unpacking, or a key that is not a string stops the command with the
file and the line: the parallel levels would be wrong without that row.
The same file is checked for a statement that adds to a name and drops
the sum, such as the msg+ in post_ready_hook. The fixture
hooks/msg-plus.py has that statement:
eb-stack stack oom --hooks tests/oom_profile_fixtures/hooks/msg-plus.py
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/msg-plus.py
warning: tests/oom_profile_fixtures/hooks/msg-plus.py:30: msg+ "to avoid out-of-memory failures during building/testing" in post_ready_hook adds to msg and drops the result, so msg does not change; msg += appends
The warning names the function and the line. It does not change a
parallel level, and the command still exits 0. msg += "...", as in
hooks/msg-plus-equals.py, appends and is not reported.
EasyBuild’s –parallel and the hook¶
post_ready_hook does not divide the level EasyBuild chose. It calls
det_parallelism() with no arguments, which counts the cores the job
can use, applies the op to that count, and keeps the result only when
it is below EasyBuild’s level. EasyBuild’s level comes from
set_parallel: --parallel (EASYBUILD_PARALLEL) in place of the
core count when it is set, capped at the easyconfig’s maxparallel and
at --max-parallel (EASYBUILD_MAX_PARALLEL, 16 by default). The bot
sets neither. stack oom --parallel N and --max-parallel N model a
job that does. --node CORES:MEMORY_MIB:TARGET prints the levels for
one node size:
eb-stack stack oom \
--hooks tests/oom_profile_fixtures/hooks/integers.py \
--parallel 12 \
--node 16:32768:x86_64/amd/zen4
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/integers.py
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4, --parallel 12
libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
job memory: libxc 7.0 unmeasured, no mib_per_job
libxc gets 8, which is 16 // 2. Half of --parallel 12 would be 6.
With --parallel 4 the hook’s 8 is not lower, so it does not apply:
eb-stack stack oom \
--hooks tests/oom_profile_fixtures/hooks/integers.py \
--parallel 4 \
--node 16:32768:x86_64/amd/zen4
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/integers.py
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4, --parallel 4
libxc: current 4 (divide_by_factor on *, max(1, 16 // 2) = 8, then --parallel 4); memory unmeasured
job memory: libxc 7.0 unmeasured, no mib_per_job
A divide factor is a fraction of the node, not of --parallel:
--parallel above the hook’s result does not change it, and
--parallel below it wins. --parallel also replaces the core count in
EasyBuild, so with --parallel 32 --max-parallel 64 a build with no
hook row runs at 32 on a 16-core node. With an easystack, the
parallel line of a warning names --parallel and the cores the hook
divided, and says when the hook’s result was not lower.
At software-layer-scripts 43df6f4 the hook reads EESSI_SOFTWARE_SUBDIR,
the software name, EasyBuild’s level, and det_parallelism(). No
environment variable there caps or overrides PARALLELISM_LIMITS.
EESSI-extend’s EESSI_EASYBUILD_HOOKS_OVERRIDE replaces the whole hooks
file; --hooks reads such a file.
A memory limit instead of a core fraction¶
A core fraction does not travel. Half of 16 cores and half of 64 cores
are different numbers of compilers, and the memory those compilers need
does not change with the core count. limit_by_memory(parallel, mib_per_job) is max(1, min(parallel, memory_mib // mib_per_job)).
memory_mib is the smallest cgroup v2 memory.max on the build
process’s cgroup path, in MiB. That is the Slurm job’s limit when the
job is the cgroup. memory.max is bytes; a value of max is unlimited
and does not count. With no cgroup v2 limit, the figure is MemTotal
from /proc/meminfo, kilobytes divided by 1024, which is what EasyBuild’s
get_total_memory() returns.
mib_per_job is the peak RSS of the worst compile, plus headroom. One
measured number then holds on every node size. data/job-memory.toml
is that table. A row with no mib_per_job is unmeasured, and the note
says so. libxc 7.0 is unmeasured. The real peak waits on a compile under
GCC 15.2 with the EESSI flags. Nothing in the table is a guess, and the
per-process GiB figures in data/build-memory.toml are not copied in:
those explain a failed job, they are not ru_maxrss of one compile.
--node prints, for each CORES:MEMORY_MIB or
CORES:MEMORY_MIB:TARGET, the parallelism the current hook leaves and
the parallelism limit_by_memory leaves. A node with no target uses
only '*' rows, not a CPU-target override and not the A64FX divide by
--job-memoryreplaces the table. With no--easystack, the report
is the whole output:
eb-stack stack oom \
--node 16:32768:x86_64/amd/zen4 \
--node 48:30016:aarch64/a64fx
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4
LLVM: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
MBX: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
PETSc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
Qt5: current 16 (no hook limit); memory unmeasured
Qt6: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
QuantumESPRESSO: current 16 (no hook limit); memory unmeasured
ROCm-LLVM: current 12 (set_maximum on *, min(16, 12)); memory unmeasured
TensorFlow: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
hipBLASLt: current 8 (set_maximum on *, min(16, 8)); memory unmeasured
jax: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
rocBLAS: current 8 (set_maximum on *, min(16, 8)); memory unmeasured
node 48 cores, 30016 MiB, aarch64/a64fx
LLVM: current 16 (divide_by_factor on *, max(1, 48 // 2) = 24, then max-parallel 16); memory unmeasured
MBX: current 1 (set_maximum on aarch64/a64fx, min(48, 1)); memory unmeasured
PETSc: current 16 (divide_by_factor on *, max(1, 48 // 2) = 24, then max-parallel 16); memory unmeasured
Qt5: current 8 (set_maximum on aarch64/a64fx, min(48, 8)); memory unmeasured
Qt6: current 8 (set_maximum on aarch64/a64fx, min(48, 8)); memory unmeasured
QuantumESPRESSO: current 6 (set_maximum on aarch64/a64fx, min(48, 6)); memory unmeasured
ROCm-LLVM: current 12 (set_maximum on *, min(48, 12)); memory unmeasured
TensorFlow: current 8 (set_maximum on aarch64/a64fx, min(48, 8)); memory unmeasured
hipBLASLt: current 8 (set_maximum on *, min(48, 8)); memory unmeasured
jax: current 16 (divide_by_factor on *, max(1, 48 // 2) = 24, then max-parallel 16); memory unmeasured
libxc: current 6 (set_maximum on aarch64/a64fx, min(48, 6)); memory unmeasured
rocBLAS: current 8 (set_maximum on *, min(48, 8)); memory unmeasured
(other builds): current 12 (divide_by_factor on aarch64/a64fx, max(1, 48 // 4)); memory unmeasured
job memory: libxc 7.0 unmeasured, no mib_per_job
On zen4, libxc’s '*' row divides 16 by 2. On A64FX the CPU-target row
overrides that and caps libxc at 6, and a build with no software row
takes the global divide by 4, which is 12. (other builds) is that
default. memory unmeasured means the table has no mib_per_job for
that name. The footer names the rows that are unmeasured. libxc 7.0 is
the one the table calls out; the other names are absent from the table,
which is the same as unmeasured.
With a number in the table, the line also prints the memory cap: the
level the hook leaves, with a memory ceiling on top,
min(current, memory_mib // mib_per_job). The memory is a whole number
of MiB, so the quotient is the same as
memory_bytes // (mib_per_job * 2**20). --mib-per-job FILE is the
same flag as --job-memory FILE. The fixture
job-memory-fixture.toml gives libxc 8192 MiB, which is arithmetic and
not a measurement. With hooks/integers.py the report is libxc alone:
eb-stack stack oom \
--hooks tests/oom_profile_fixtures/hooks/integers.py \
--mib-per-job tests/oom_profile_fixtures/job-memory-fixture.toml \
--node 16:32768:x86_64/amd/zen4 \
--node 16:131072:x86_64/amd/zen4 \
--node 16:4096:x86_64/amd/zen4
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/integers.py
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4
libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory cap 4 (min(8, 32768 // 8192)); memory 4 (max(1, min(16, 32768 // 8192)))
node 16 cores, 131072 MiB, x86_64/amd/zen4
libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory cap 8 (min(8, 131072 // 8192)); memory 16 (max(1, min(16, 131072 // 8192)))
node 16 cores, 4096 MiB, x86_64/amd/zen4
libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory cap 0 (min(8, 4096 // 8192)), not one 8192 MiB job fits; memory 1 (max(1, min(16, 4096 // 8192)))
job memory: every package in the table has an mib_per_job
On 32 GiB the memory allows four 8192 MiB jobs and the hook allows 8,
so the memory cap is 4. On 128 GiB the memory allows 16, and the cap
stays at the hook’s 8; limit_by_memory in place of the hook would
give 16. On 4 GiB not one job fits. limit_by_memory still says 1,
because it never goes below one job. The memory cap is 0: one compile
already needs more than the node has.
A cap of 8 or a reduction of 8 for libxc¶
Two forms of libxc’s '*' row in PARALLELISM_LIMITS have come up in
the review of software-layer-scripts pull request 337. Each fixture is
hooks/integers.py with that row in place of (divide_by_factor, 2),
and the A64FX row stays at (set_maximum, 6):
hooks/set-maximum-8.pyhas a cap of 8,(set_maximum, 8), the form proposed in review.hooks/divide-8.pyhas a reduction of 8,(divide_by_factor, 8), as asked in the review of 337 (issuecomment-6599147327).
Compare node sizes under the cap of 8:
eb-stack stack oom \
--hooks tests/oom_profile_fixtures/hooks/set-maximum-8.py \
--node 16:32768:x86_64/amd/zen4 \
--node 48:30016:aarch64/a64fx \
--node 64:262144:x86_64/amd/zen4 \
--node 12:24576:x86_64/amd/zen4
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/set-maximum-8.py
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4
libxc: current 8 (set_maximum on *, min(16, 8)); memory unmeasured
node 48 cores, 30016 MiB, aarch64/a64fx
libxc: current 6 (set_maximum on aarch64/a64fx, min(48, 6)); memory unmeasured
(other builds): current 12 (divide_by_factor on aarch64/a64fx, max(1, 48 // 4)); memory unmeasured
node 64 cores, 262144 MiB, x86_64/amd/zen4
libxc: current 8 (set_maximum on *, min(64, 8)); memory unmeasured
node 12 cores, 24576 MiB, x86_64/amd/zen4
libxc: current 8 (set_maximum on *, min(12, 8)); memory unmeasured
job memory: libxc 7.0 unmeasured, no mib_per_job
With hooks/integers.py the same four nodes give libxc 8, 6, 16 and 6.
A cap is a number of jobs, so it does not grow with the node: on 64
cores it gives 8, where 64 // 2 is 32 and max-parallel holds that at
It does not shrink either: on 12 cores it gives 8, where
12 // 2
is 6. The two rows agree on 16 and 17 cores. Every node in the bot resource table has 16 cores except A64FX, which keeps its own row, so on the bot the cap leaves libxc where it is. Pull request 1772 still warns on the 32 GiB zen4 node:
eb-stack stack oom \
--arch x86_64/amd/zen4 \
--hooks tests/oom_profile_fixtures/hooks/set-maximum-8.py \
--easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/set-maximum-8.py
libxc-7.0.0-GCC-15.2.0.eb may run out of memory on x86_64/amd/zen4 (eessi-bot-mc-aws amd-zen4): 8 x 6.4 GiB = 51.2 GiB, the job has 32.0 GiB (software-layer #1772, software-layer-scripts #337)
parallel 8: 16 cores, eb_hooks.py caps libxc at 8
why: cc1 on the generated mgga_x_*.c sources; with GCC 15.2.0 on zen4, 8 and then 5 compilers ran out of a 32 GiB job
remedy: maxparallel = 4 in libxc-7.0.0-GCC-15.2.0.eb, or a libxc limit of 4 for x86_64/amd/zen4 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 6.4 GiB per process is a lower bound, so 4 may still be too many
remedy: build x86_64/amd/zen4 on a node with more memory: eessi-bot-surf amd-zen4 (16 cores, 180.0 GiB, accel nvidia/cc90)
remedy: a libxc divider of 3 left 5 cores and ran out too; a reduction of 8, as asked in the review of software-layer-scripts #337 (issuecomment-6599147327), is 2 cores on the AWS build cluster
oom=1 fits=1 builds=5 nodes=2
The estimate is the one in the first example. Only the parallel line
changes, and it names the cap in place of the divide. The reduction of
8 in hooks/divide-8.py leaves 2 cores on these nodes:
eb-stack stack oom \
--arch x86_64/amd/zen4 \
--hooks tests/oom_profile_fixtures/hooks/divide-8.py \
--easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/divide-8.py
oom=0 fits=2 builds=5 nodes=2
max(1, 16 // 8) is 2, so libxc fits on both zen4 nodes. On 64 cores
the same row gives 8.
Measure the worst compile¶
mib_per_job comes from a compiler launcher, not from a guess.
scripts/compile_rss.py is the launcher. CMake runs
CMAKE_C_COMPILER_LAUNCHER in front of the compiler. A CC wrapper is
the same shape, the script and then the compiler. Each invocation
appends one JSON line to $EB_STACK_COMPILE_RSS_LOG (default
compile-rss.jsonl in the working directory). The line is
getrusage(RUSAGE_CHILDREN).ru_maxrss after the compiler exits. On
Linux that field is kilobytes, and the log calls it ru_maxrss_kb. A
lock keeps a parallel build from tearing a line.
Build under the EESSI flags. <target> is the -march the bot uses
for that CPU target (znver4 on zen4, and the matching name on the
others):
export EB_STACK_COMPILE_RSS_LOG="$PWD/compile-rss.jsonl"
export CFLAGS="-O2 -ftree-vectorize -march=<target> -fno-math-errno"
export CXXFLAGS="$CFLAGS"
export CMAKE_C_COMPILER_LAUNCHER="$PWD/scripts/compile_rss.py"
export CMAKE_CXX_COMPILER_LAUNCHER="$PWD/scripts/compile_rss.py"
# or, when the build sets CC itself:
# export CC="$PWD/scripts/compile_rss.py gcc"
Then run the build. The summary picks the worst compile and suggests
mib_per_job: the peak rounded up to a whole MiB, plus headroom of one
quarter (--headroom 0.25), rounded up again. Copy that number into
data/job-memory.toml yourself, with a note that says where the log
came from. The suggestion is arithmetic on the log. It does not name a
package, and it does not write the table.
The fixture below is not a measurement of libxc, or of any package. It only shows the summary. 4096 kilobytes is 4 MiB, and one quarter of headroom makes 5:
python3 scripts/compile_rss.py --summarize \
tests/oom_profile_fixtures/compile-rss.jsonl
compiles: 2
worst_ru_maxrss_kb: 4096
worst_command: gcc -O2 -ftree-vectorize -march=znver4 -fno-math-errno -c sample.c
worst_exit: 0
worst_mib: 4
headroom: 1/4
suggested_mib_per_job: 5
note: suggested_mib_per_job is the worst peak rounded up to MiB, plus headroom, rounded up; it is not a measurement of a named package
libxc’s row stays unmeasured until that launcher has been run on a libxc 7.0.0 build with those flags. Do not fill the number in before then.
Limits¶
Software with no entry in
data/build-memory.tomlis not checked. The count line shows how many pairs had a profile:fitsplusoom.A target with no node in the bot resource table gets a note and is not checked.
The model is linear in the parallel level. It does not model link steps, test steps that run with a different parallel level, or memory that other jobs on the node use. It does not model
ulimit -u, whichdet_parallelismalso consults.--hooksreads thePARALLELISM_LIMITSliteral. It does not runpost_ready_hook; the rules that pick a row are the ones described under Where the numbers come from. Of the rest of the file it checks only for a statement that drops a sum. Other mistakes in the hooks are not found.limit_by_memoryand the memory cap do not replace the estimate. The estimate follows the hooks that are ineb_hooks.py. The--nodereport is the comparison, and the warnings for an easystack do not print the cap. A package with no measuredmib_per_jobstays unmeasured.The MCP tool
eb_stack_oomreturns the same lines and each warning as JSON.