Warn before a bot job runs out of memory

What it does: stack oom reads an easystack as a plan. For each entry and each EESSI bot build node, it estimates the peak build memory at the parallel level the bot will use. When that estimate reaches the job’s memory, it prints a warning with remedies. The remedies are text only. The command never edits a recipe or a hook, and it always exits 0.

Why you’d use it: a bot job that runs out of memory fails late and quietly. Software-layer pull request 1772 added libxc-7.0.0-GCC-15.2.0.eb. The build was killed for lack of memory on zen4 twice at 8 processes on a 32 GiB job, and again at 5 after a hook change. The same thing happened to Highway in #1692, to hipBLASLt and rocBLAS in #1549, and to PETSc in #1608. Run the check before you open the pull request or ask the bot to build.

eb-stack stack oom \
  --easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml

The command prints:

libxc-7.0.0-GCC-15.2.0.eb may run out of memory on x86_64/amd/zen4 (eessi-bot-mc-aws amd-zen4): 8 x 6.4 GiB = 51.2 GiB, the job has 32.0 GiB (software-layer #1772, software-layer-scripts #337)
  parallel 8: 16 cores, eb_hooks.py divides libxc by 2
  why: cc1 on the generated mgga_x_*.c sources; with GCC 15.2.0 on zen4, 8 and then 5 compilers ran out of a 32 GiB job
  remedy: maxparallel = 4 in libxc-7.0.0-GCC-15.2.0.eb, or a libxc limit of 4 for x86_64/amd/zen4 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 6.4 GiB per process is a lower bound, so 4 may still be too many
  remedy: build x86_64/amd/zen4 on a node with more memory: eessi-bot-surf amd-zen4 (16 cores, 180.0 GiB, accel nvidia/cc90)
  remedy: a libxc divider of 3 left 5 cores and ran out too; a reduction of 8, as asked in the review of software-layer-scripts #337 (issuecomment-6599147327), is 2 cores on the AWS build cluster
oom=1 fits=14 builds=5 nodes=15

How to read it:

  • The first line names the entry, the target, the bot instance and the node, and the estimate against the job’s memory. The pull request numbers are where the figure was measured.

  • parallel 8 is how the bot gets to 8. The node has 16 cores, and the EESSI post_ready_hook halves libxc.

  • why says what used the memory in the real job.

  • Each remedy is a separate way out: a lower maxparallel or hook limit, a node with more memory for the same target, and what the maintainers did or asked for.

  • The count line says one entry and node pair may run out of memory, 14 pairs have a profile and fit, the plan has 5 builds, and 15 nodes were checked. Without --arch, every node in the bot resource table is checked.

A dependency the easystack does not name

With --easyconfigs, the command closes the plan’s dependencies, so a dependency that is not installed yet is checked too. Pull request 1692 added only eOn. Highway came in as a dependency and failed on aarch64/generic:

eb-stack stack oom \
  --arch aarch64/generic \
  --easystack tests/oom_profile_fixtures/easystacks/pr1692-eessi-2026.06-eb-5.4.0-2026.1.yml \
  --easyconfigs tests/oom_profile_fixtures/easyconfigs
Highway-1.4.0-GCCcore-15.2.0.eb may run out of memory on aarch64/generic (eessi-bot-mc-aws generic): 16 x 1.9 GiB = 30.4 GiB, the job has 29.8 GiB (software-layer #1692, software-layer-scripts #317) (required by eOn-3.6.0-foss-2026.1.eb)
  parallel 16: 16 cores, no eb_hooks.py limit for Highway
  why: compiling the hwy/tests sources exhausts virtual memory
  remedy: maxparallel = 15 in Highway-1.4.0-GCCcore-15.2.0.eb, or a Highway limit of 15 for aarch64/generic in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 1.9 GiB per process is a lower bound, so 15 may still be too many
  remedy: no other aarch64/generic node in the bot resource table has room for it
  remedy: software-layer-scripts #317 halves Highway on every target and keeps A64FX at a quarter
oom=1 fits=0 builds=2 nodes=1

(required by eOn-3.6.0-foss-2026.1.eb) says which easystack entry pulled Highway in. Add --installed with one name/version-toolchain module per line to skip what the target already has.

When fewer cores do not help

Some builds need a fixed amount of memory whatever the parallel level. Pull request 1549 built rocBLAS for ROCm 6.4.1, and hipBLASLt came in as its dependency:

eb-stack stack oom \
  --arch x86_64/amd/zen2 \
  --easystack tests/oom_profile_fixtures/easystacks/pr1549-eessi-2025.06-eb-5.3.1-2025a.yml \
  --easyconfigs tests/oom_profile_fixtures/easyconfigs
hipBLASLt-0.12.1-rocm-compilers-19.0.0-ROCm-6.4.1.eb may run out of memory on x86_64/amd/zen2 (eessi-bot-mc-aws amd-zen2): 90.0 GiB at any parallel level, the job has 27.9 GiB (software-layer-scripts #281, software-layer #1549) (required by rocBLAS-4.4.0-rocm-compilers-19.0.0-ROCm-6.4.1.eb)
  parallel 8: 16 cores, eb_hooks.py caps hipBLASLt at 8
  why: Tensile kernel generation for gfx942 holds about 90 GiB whatever the parallel level
  remedy: a lower parallel level does not help: one process already needs 90.0 GiB
  remedy: no other x86_64/amd/zen2 node in the bot resource table has room for it
  remedy: the hipBLASLt and rocBLAS jobs ran on the larger -xl build nodes, through a temporary bot config (software-layer-scripts #302)
rocBLAS-4.4.0-rocm-compilers-19.0.0-ROCm-6.4.1.eb may run out of memory on x86_64/amd/zen2 (eessi-bot-mc-aws amd-zen2): 50.0 GiB at any parallel level, the job has 27.9 GiB (software-layer-scripts #302, software-layer #1549)
  parallel 8: 16 cores, eb_hooks.py caps rocBLAS at 8
  why: Tensile kernel generation for gfx942 needs more than 50-60 GB
  remedy: a lower parallel level does not help: one process already needs 50.0 GiB
  remedy: no other x86_64/amd/zen2 node in the bot resource table has room for it
  remedy: the hipBLASLt and rocBLAS jobs ran on the larger -xl build nodes, through a temporary bot config (software-layer-scripts #302)
oom=2 fits=0 builds=2 nodes=1

The estimate says at any parallel level, so the only remedy is a bigger node.

Check a hook change

The parallel level comes from the hook limits in data/eessi-bot-resources.toml. Pass --resources to try other limits or other nodes, or --hooks to read the limits from an eb_hooks.py (see Read the limits from eb:sub:`hooks.py`_). Pull request 1608 ran out of memory in the PETSc test step on zen5 before software-layer-scripts #298 halved PETSc. With the current hooks, PETSc fits:

eb-stack stack oom \
  --arch x86_64/amd/zen5 \
  --easystack tests/oom_profile_fixtures/easystacks/pr1608-eessi-2025.06-eb-5.4.0-2025b.yml \
  --easyconfigs tests/oom_profile_fixtures/easyconfigs
oom=0 fits=1 builds=3 nodes=1

The fixture resources-before-298.toml has the zen5 node and the hooks as they were before #298:

eb-stack stack oom \
  --resources tests/oom_profile_fixtures/resources-before-298.toml \
  --easystack tests/oom_profile_fixtures/easystacks/pr1608-eessi-2025.06-eb-5.4.0-2025b.yml \
  --easyconfigs tests/oom_profile_fixtures/easyconfigs
PETSc-3.24.0-foss-2025b.eb may run out of memory on x86_64/amd/zen5 (eessi-bot-aws-eu-south amd-zen5): 16 x 2.0 GiB = 32.0 GiB, the job has 31.5 GiB (software-layer #1608, software-layer-scripts #298) (required by slepc4py-3.24.0-foss-2025b.eb)
  parallel 16: 16 cores, no eb_hooks.py limit for PETSc
  why: the test step runs make -j <parallel> test, and that is what ran out
  remedy: maxparallel = 15 in PETSc-3.24.0-foss-2025b.eb, or a PETSc limit of 15 for x86_64/amd/zen5 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 2.0 GiB per process is a lower bound, so 15 may still be too many
  remedy: no other x86_64/amd/zen5 node in the bot resource table has room for it
  remedy: software-layer-scripts #298 halves PETSc on every target
oom=1 fits=0 builds=3 nodes=1

Where the numbers come from

Two data files hold the numbers, and every row cites a real pull request or bot configuration.

  • data/build-memory.toml holds the peak build memory per software and version family. An entry may be narrowed to a toolchain family and a target. The estimate is base_gib + per_process_gib * parallel. Each entry lists the bot jobs it came from, with the parallel level, the job’s memory, and whether it ran out. The table is refused when loading if it does not reproduce every one of those jobs, so a new figure cannot contradict an old observation. A figure marked lower_bound is the least that explains a failure. The real peak may be higher.

  • data/eessi-bot-resources.toml holds each bot build node: the instance, the node, the target, the cores and the job’s memory. It also holds EasyBuild’s default maximum parallel level of 16 and the PARALLELISM_LIMITS table from the EESSI eb_hooks.py.

The parallel level is the smallest of these: the node’s cores, or --parallel in their place; EasyBuild’s maximum; the hook limit applied to the cores; and a top-level maxparallel in the recipe. The hook ops are the ones in eb_hooks.py. divide_by_factor(parallel, factor) is max(1, parallel // factor). set_maximum(parallel, n) is min(parallel, n). A software name uses its limit for the CPU target, and that key overrides its '*' limit. Only software with no hook entry gets the target’s own limit, which divides by 4 on A64FX. The hook applies the op to the cores the job can use, and keeps the result only when it is lower than the parallel level EasyBuild already chose.

When more than one entry matches a build, the highest estimate wins.

divide in the hook table is an integer in eb_hooks.py. A factor that is not an integer makes Python’s // return a float: 16 // 2.5 is 6.0, and max keeps that float when it is at least 1. A whole number written as a float does the same: 16 // 2.0 is 8.0. The command warns, and the estimate uses the numeric floor. Write the factor as an integer.

In post_ready_hook the log line is two statements:

msg = "limiting parallelism to %s (was %s, derived parallelism %s) for %s on %s "
msg+ "to avoid out-of-memory failures during building/testing"

The second statement adds nothing. msg + "..." is an expression, and the result is thrown away. The operator that appends is +=. The line that prints stops after the CPU target. This is upstream eb_hooks.py. eb-stack does not patch it. --hooks reports it with its line (see Read the limits from eb:sub:`hooks.py`_).

A divide factor that is not an integer

The maintainers asked for a libxc divider of 2.5 or 3, which is 6 or 5 cores on the 16-core AWS build nodes. 3 is an integer. 2.5 is not, and eb_hooks.py would store a float. The fixture resources-divide-2.5.toml is that factor, not the integer 2 the hooks ship:

eb-stack stack oom \
  --arch x86_64/amd/zen4 \
  --resources tests/oom_profile_fixtures/resources-divide-2.5.toml \
  --easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml
warning: PARALLELISM_LIMITS divide_by_factor for libxc on * uses 2.5, which is not an integer; eb_hooks.py computes max(1, parallel // factor), and 16 // 2.5 is 6.0, a float. Use an integer factor.
libxc-7.0.0-GCC-15.2.0.eb may run out of memory on x86_64/amd/zen4 (eessi-bot-mc-aws amd-zen4): 6 x 6.4 GiB = 38.4 GiB, the job has 32.0 GiB (software-layer #1772, software-layer-scripts #337)
  parallel 6: 16 cores, eb_hooks.py divides libxc by 2.5 (16 // 2.5 is 6.0, a float)
  why: cc1 on the generated mgga_x_*.c sources; with GCC 15.2.0 on zen4, 8 and then 5 compilers ran out of a 32 GiB job
  remedy: maxparallel = 4 in libxc-7.0.0-GCC-15.2.0.eb, or a libxc limit of 4 for x86_64/amd/zen4 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 6.4 GiB per process is a lower bound, so 4 may still be too many
  remedy: no other x86_64/amd/zen4 node in the bot resource table has room for it
  remedy: a libxc divider of 3 left 5 cores and ran out too; a reduction of 8, as asked in the review of software-layer-scripts #337 (issuecomment-6599147327), is 2 cores on the AWS build cluster
oom=1 fits=0 builds=5 nodes=1

The first warning is the float. The estimate then uses 6, and 6 times 6.4 GiB is 38.4 GiB, more than the 32 GiB zen4 job. The divider of 3 ran out at 5 in bot job 221273 of #1772, so 6 does not fit either. The fixture has only the AWS zen4 node, so no other node has room. The command still exits 0.

Read the limits from ebhooks.py

The hook rows in data/eessi-bot-resources.toml are a copy of PARALLELISM_LIMITS. --hooks reads that dict from an eb_hooks.py instead, so a hook change can be checked before it is merged. The file is parsed as Python, and nothing in it runs. The fixture hooks/divide-2.5.py has the layout of the real file and the libxc divider of 2.5:

eb-stack stack oom \
  --arch x86_64/amd/zen4 \
  --hooks tests/oom_profile_fixtures/hooks/divide-2.5.py \
  --easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/divide-2.5.py
warning: tests/oom_profile_fixtures/hooks/divide-2.5.py:20: PARALLELISM_LIMITS divide_by_factor for libxc on * uses 2.5, which is not an integer; eb_hooks.py computes max(1, parallel // factor), and 16 // 2.5 is 6.0, a float. Use an integer factor.
libxc-7.0.0-GCC-15.2.0.eb may run out of memory on x86_64/amd/zen4 (eessi-bot-mc-aws amd-zen4): 6 x 6.4 GiB = 38.4 GiB, the job has 32.0 GiB (software-layer #1772, software-layer-scripts #337)
  parallel 6: 16 cores, eb_hooks.py divides libxc by 2.5 (16 // 2.5 is 6.0, a float)
  why: cc1 on the generated mgga_x_*.c sources; with GCC 15.2.0 on zen4, 8 and then 5 compilers ran out of a 32 GiB job
  remedy: maxparallel = 4 in libxc-7.0.0-GCC-15.2.0.eb, or a libxc limit of 4 for x86_64/amd/zen4 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 6.4 GiB per process is a lower bound, so 4 may still be too many
  remedy: build x86_64/amd/zen4 on a node with more memory: eessi-bot-surf amd-zen4 (16 cores, 180.0 GiB, accel nvidia/cc90)
  remedy: a libxc divider of 3 left 5 cores and ran out too; a reduction of 8, as asked in the review of software-layer-scripts #337 (issuecomment-6599147327), is 2 cores on the AWS build cluster
oom=1 fits=1 builds=5 nodes=2

The first line says how many rows were read, and from which file. The first warning names the line of the (divide_by_factor, 2.5) tuple. The nodes still come from the bot resource table, so both zen4 nodes are checked, and the 180 GiB node has room for 6. With every factor an integer there is no warning: line:

eb-stack stack oom --hooks tests/oom_profile_fixtures/hooks/integers.py
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/integers.py

The libxc and A64FX rows of integers.py are the ones in the bot resource table, so --node (see A memory limit instead of a core fraction) prints the same parallel levels with and without it. For the real hooks, pass the file from a software-layer-scripts checkout, --hooks software-layer-scripts/eb_hooks.py.

A key is a string or a module-level string constant such as CPU_TARGET_A64FX. A key the dict repeats keeps its last value, as in Python. An op other than divide_by_factor or set_maximum, a \*\* unpacking, or a key that is not a string stops the command with the file and the line: the parallel levels would be wrong without that row.

The same file is checked for a statement that adds to a name and drops the sum, such as the msg+ in post_ready_hook. The fixture hooks/msg-plus.py has that statement:

eb-stack stack oom --hooks tests/oom_profile_fixtures/hooks/msg-plus.py
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/msg-plus.py
warning: tests/oom_profile_fixtures/hooks/msg-plus.py:30: msg+ "to avoid out-of-memory failures during building/testing" in post_ready_hook adds to msg and drops the result, so msg does not change; msg += appends

The warning names the function and the line. It does not change a parallel level, and the command still exits 0. msg += "...", as in hooks/msg-plus-equals.py, appends and is not reported.

EasyBuild’s –parallel and the hook

post_ready_hook does not divide the level EasyBuild chose. It calls det_parallelism() with no arguments, which counts the cores the job can use, applies the op to that count, and keeps the result only when it is below EasyBuild’s level. EasyBuild’s level comes from set_parallel: --parallel (EASYBUILD_PARALLEL) in place of the core count when it is set, capped at the easyconfig’s maxparallel and at --max-parallel (EASYBUILD_MAX_PARALLEL, 16 by default). The bot sets neither. stack oom --parallel N and --max-parallel N model a job that does. --node CORES:MEMORY_MIB:TARGET prints the levels for one node size:

eb-stack stack oom \
  --hooks tests/oom_profile_fixtures/hooks/integers.py \
  --parallel 12 \
  --node 16:32768:x86_64/amd/zen4
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/integers.py
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4, --parallel 12
  libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
job memory: libxc 7.0 unmeasured, no mib_per_job

libxc gets 8, which is 16 // 2. Half of --parallel 12 would be 6. With --parallel 4 the hook’s 8 is not lower, so it does not apply:

eb-stack stack oom \
  --hooks tests/oom_profile_fixtures/hooks/integers.py \
  --parallel 4 \
  --node 16:32768:x86_64/amd/zen4
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/integers.py
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4, --parallel 4
  libxc: current 4 (divide_by_factor on *, max(1, 16 // 2) = 8, then --parallel 4); memory unmeasured
job memory: libxc 7.0 unmeasured, no mib_per_job

A divide factor is a fraction of the node, not of --parallel: --parallel above the hook’s result does not change it, and --parallel below it wins. --parallel also replaces the core count in EasyBuild, so with --parallel 32 --max-parallel 64 a build with no hook row runs at 32 on a 16-core node. With an easystack, the parallel line of a warning names --parallel and the cores the hook divided, and says when the hook’s result was not lower.

At software-layer-scripts 43df6f4 the hook reads EESSI_SOFTWARE_SUBDIR, the software name, EasyBuild’s level, and det_parallelism(). No environment variable there caps or overrides PARALLELISM_LIMITS. EESSI-extend’s EESSI_EASYBUILD_HOOKS_OVERRIDE replaces the whole hooks file; --hooks reads such a file.

A memory limit instead of a core fraction

A core fraction does not travel. Half of 16 cores and half of 64 cores are different numbers of compilers, and the memory those compilers need does not change with the core count. limit_by_memory(parallel, mib_per_job) is max(1, min(parallel, memory_mib // mib_per_job)). memory_mib is the smallest cgroup v2 memory.max on the build process’s cgroup path, in MiB. That is the Slurm job’s limit when the job is the cgroup. memory.max is bytes; a value of max is unlimited and does not count. With no cgroup v2 limit, the figure is MemTotal from /proc/meminfo, kilobytes divided by 1024, which is what EasyBuild’s get_total_memory() returns.

mib_per_job is the peak RSS of the worst compile, plus headroom. One measured number then holds on every node size. data/job-memory.toml is that table. A row with no mib_per_job is unmeasured, and the note says so. libxc 7.0 is unmeasured. The real peak waits on a compile under GCC 15.2 with the EESSI flags. Nothing in the table is a guess, and the per-process GiB figures in data/build-memory.toml are not copied in: those explain a failed job, they are not ru_maxrss of one compile.

--node prints, for each CORES:MEMORY_MIB or CORES:MEMORY_MIB:TARGET, the parallelism the current hook leaves and the parallelism limit_by_memory leaves. A node with no target uses only '*' rows, not a CPU-target override and not the A64FX divide by

  1. --job-memory replaces the table. With no --easystack, the report

is the whole output:

eb-stack stack oom \
  --node 16:32768:x86_64/amd/zen4 \
  --node 48:30016:aarch64/a64fx
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4
  LLVM: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
  MBX: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
  PETSc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
  Qt5: current 16 (no hook limit); memory unmeasured
  Qt6: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
  QuantumESPRESSO: current 16 (no hook limit); memory unmeasured
  ROCm-LLVM: current 12 (set_maximum on *, min(16, 12)); memory unmeasured
  TensorFlow: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
  hipBLASLt: current 8 (set_maximum on *, min(16, 8)); memory unmeasured
  jax: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
  libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory unmeasured
  rocBLAS: current 8 (set_maximum on *, min(16, 8)); memory unmeasured
node 48 cores, 30016 MiB, aarch64/a64fx
  LLVM: current 16 (divide_by_factor on *, max(1, 48 // 2) = 24, then max-parallel 16); memory unmeasured
  MBX: current 1 (set_maximum on aarch64/a64fx, min(48, 1)); memory unmeasured
  PETSc: current 16 (divide_by_factor on *, max(1, 48 // 2) = 24, then max-parallel 16); memory unmeasured
  Qt5: current 8 (set_maximum on aarch64/a64fx, min(48, 8)); memory unmeasured
  Qt6: current 8 (set_maximum on aarch64/a64fx, min(48, 8)); memory unmeasured
  QuantumESPRESSO: current 6 (set_maximum on aarch64/a64fx, min(48, 6)); memory unmeasured
  ROCm-LLVM: current 12 (set_maximum on *, min(48, 12)); memory unmeasured
  TensorFlow: current 8 (set_maximum on aarch64/a64fx, min(48, 8)); memory unmeasured
  hipBLASLt: current 8 (set_maximum on *, min(48, 8)); memory unmeasured
  jax: current 16 (divide_by_factor on *, max(1, 48 // 2) = 24, then max-parallel 16); memory unmeasured
  libxc: current 6 (set_maximum on aarch64/a64fx, min(48, 6)); memory unmeasured
  rocBLAS: current 8 (set_maximum on *, min(48, 8)); memory unmeasured
  (other builds): current 12 (divide_by_factor on aarch64/a64fx, max(1, 48 // 4)); memory unmeasured
job memory: libxc 7.0 unmeasured, no mib_per_job

On zen4, libxc’s '*' row divides 16 by 2. On A64FX the CPU-target row overrides that and caps libxc at 6, and a build with no software row takes the global divide by 4, which is 12. (other builds) is that default. memory unmeasured means the table has no mib_per_job for that name. The footer names the rows that are unmeasured. libxc 7.0 is the one the table calls out; the other names are absent from the table, which is the same as unmeasured.

With a number in the table, the line also prints the memory cap: the level the hook leaves, with a memory ceiling on top, min(current, memory_mib // mib_per_job). The memory is a whole number of MiB, so the quotient is the same as memory_bytes // (mib_per_job * 2**20). --mib-per-job FILE is the same flag as --job-memory FILE. The fixture job-memory-fixture.toml gives libxc 8192 MiB, which is arithmetic and not a measurement. With hooks/integers.py the report is libxc alone:

eb-stack stack oom \
  --hooks tests/oom_profile_fixtures/hooks/integers.py \
  --mib-per-job tests/oom_profile_fixtures/job-memory-fixture.toml \
  --node 16:32768:x86_64/amd/zen4 \
  --node 16:131072:x86_64/amd/zen4 \
  --node 16:4096:x86_64/amd/zen4
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/integers.py
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4
  libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory cap 4 (min(8, 32768 // 8192)); memory 4 (max(1, min(16, 32768 // 8192)))
node 16 cores, 131072 MiB, x86_64/amd/zen4
  libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory cap 8 (min(8, 131072 // 8192)); memory 16 (max(1, min(16, 131072 // 8192)))
node 16 cores, 4096 MiB, x86_64/amd/zen4
  libxc: current 8 (divide_by_factor on *, max(1, 16 // 2)); memory cap 0 (min(8, 4096 // 8192)), not one 8192 MiB job fits; memory 1 (max(1, min(16, 4096 // 8192)))
job memory: every package in the table has an mib_per_job

On 32 GiB the memory allows four 8192 MiB jobs and the hook allows 8, so the memory cap is 4. On 128 GiB the memory allows 16, and the cap stays at the hook’s 8; limit_by_memory in place of the hook would give 16. On 4 GiB not one job fits. limit_by_memory still says 1, because it never goes below one job. The memory cap is 0: one compile already needs more than the node has.

A cap of 8 or a reduction of 8 for libxc

Two forms of libxc’s '*' row in PARALLELISM_LIMITS have come up in the review of software-layer-scripts pull request 337. Each fixture is hooks/integers.py with that row in place of (divide_by_factor, 2), and the A64FX row stays at (set_maximum, 6):

  • hooks/set-maximum-8.py has a cap of 8, (set_maximum, 8), the form proposed in review.

  • hooks/divide-8.py has a reduction of 8, (divide_by_factor, 8), as asked in the review of 337 (issuecomment-6599147327).

Compare node sizes under the cap of 8:

eb-stack stack oom \
  --hooks tests/oom_profile_fixtures/hooks/set-maximum-8.py \
  --node 16:32768:x86_64/amd/zen4 \
  --node 48:30016:aarch64/a64fx \
  --node 64:262144:x86_64/amd/zen4 \
  --node 12:24576:x86_64/amd/zen4
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/set-maximum-8.py
parallelism under the current eb_hooks.py limit and under limit_by_memory
node 16 cores, 32768 MiB, x86_64/amd/zen4
  libxc: current 8 (set_maximum on *, min(16, 8)); memory unmeasured
node 48 cores, 30016 MiB, aarch64/a64fx
  libxc: current 6 (set_maximum on aarch64/a64fx, min(48, 6)); memory unmeasured
  (other builds): current 12 (divide_by_factor on aarch64/a64fx, max(1, 48 // 4)); memory unmeasured
node 64 cores, 262144 MiB, x86_64/amd/zen4
  libxc: current 8 (set_maximum on *, min(64, 8)); memory unmeasured
node 12 cores, 24576 MiB, x86_64/amd/zen4
  libxc: current 8 (set_maximum on *, min(12, 8)); memory unmeasured
job memory: libxc 7.0 unmeasured, no mib_per_job

With hooks/integers.py the same four nodes give libxc 8, 6, 16 and 6. A cap is a number of jobs, so it does not grow with the node: on 64 cores it gives 8, where 64 // 2 is 32 and max-parallel holds that at

  1. It does not shrink either: on 12 cores it gives 8, where 12 // 2

is 6. The two rows agree on 16 and 17 cores. Every node in the bot resource table has 16 cores except A64FX, which keeps its own row, so on the bot the cap leaves libxc where it is. Pull request 1772 still warns on the 32 GiB zen4 node:

eb-stack stack oom \
  --arch x86_64/amd/zen4 \
  --hooks tests/oom_profile_fixtures/hooks/set-maximum-8.py \
  --easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/set-maximum-8.py
libxc-7.0.0-GCC-15.2.0.eb may run out of memory on x86_64/amd/zen4 (eessi-bot-mc-aws amd-zen4): 8 x 6.4 GiB = 51.2 GiB, the job has 32.0 GiB (software-layer #1772, software-layer-scripts #337)
  parallel 8: 16 cores, eb_hooks.py caps libxc at 8
  why: cc1 on the generated mgga_x_*.c sources; with GCC 15.2.0 on zen4, 8 and then 5 compilers ran out of a 32 GiB job
  remedy: maxparallel = 4 in libxc-7.0.0-GCC-15.2.0.eb, or a libxc limit of 4 for x86_64/amd/zen4 in PARALLELISM_LIMITS in software-layer-scripts eb_hooks.py; 6.4 GiB per process is a lower bound, so 4 may still be too many
  remedy: build x86_64/amd/zen4 on a node with more memory: eessi-bot-surf amd-zen4 (16 cores, 180.0 GiB, accel nvidia/cc90)
  remedy: a libxc divider of 3 left 5 cores and ran out too; a reduction of 8, as asked in the review of software-layer-scripts #337 (issuecomment-6599147327), is 2 cores on the AWS build cluster
oom=1 fits=1 builds=5 nodes=2

The estimate is the one in the first example. Only the parallel line changes, and it names the cap in place of the divide. The reduction of 8 in hooks/divide-8.py leaves 2 cores on these nodes:

eb-stack stack oom \
  --arch x86_64/amd/zen4 \
  --hooks tests/oom_profile_fixtures/hooks/divide-8.py \
  --easystack tests/oom_profile_fixtures/easystacks/pr1772-eessi-2026.06-eb-5.4.0-2026.1.yml
hooks: 3 PARALLELISM_LIMITS rows from tests/oom_profile_fixtures/hooks/divide-8.py
oom=0 fits=2 builds=5 nodes=2

max(1, 16 // 8) is 2, so libxc fits on both zen4 nodes. On 64 cores the same row gives 8.

Measure the worst compile

mib_per_job comes from a compiler launcher, not from a guess. scripts/compile_rss.py is the launcher. CMake runs CMAKE_C_COMPILER_LAUNCHER in front of the compiler. A CC wrapper is the same shape, the script and then the compiler. Each invocation appends one JSON line to $EB_STACK_COMPILE_RSS_LOG (default compile-rss.jsonl in the working directory). The line is getrusage(RUSAGE_CHILDREN).ru_maxrss after the compiler exits. On Linux that field is kilobytes, and the log calls it ru_maxrss_kb. A lock keeps a parallel build from tearing a line.

Build under the EESSI flags. <target> is the -march the bot uses for that CPU target (znver4 on zen4, and the matching name on the others):

export EB_STACK_COMPILE_RSS_LOG="$PWD/compile-rss.jsonl"
export CFLAGS="-O2 -ftree-vectorize -march=<target> -fno-math-errno"
export CXXFLAGS="$CFLAGS"
export CMAKE_C_COMPILER_LAUNCHER="$PWD/scripts/compile_rss.py"
export CMAKE_CXX_COMPILER_LAUNCHER="$PWD/scripts/compile_rss.py"
# or, when the build sets CC itself:
# export CC="$PWD/scripts/compile_rss.py gcc"

Then run the build. The summary picks the worst compile and suggests mib_per_job: the peak rounded up to a whole MiB, plus headroom of one quarter (--headroom 0.25), rounded up again. Copy that number into data/job-memory.toml yourself, with a note that says where the log came from. The suggestion is arithmetic on the log. It does not name a package, and it does not write the table.

The fixture below is not a measurement of libxc, or of any package. It only shows the summary. 4096 kilobytes is 4 MiB, and one quarter of headroom makes 5:

python3 scripts/compile_rss.py --summarize \
  tests/oom_profile_fixtures/compile-rss.jsonl
compiles: 2
worst_ru_maxrss_kb: 4096
worst_command: gcc -O2 -ftree-vectorize -march=znver4 -fno-math-errno -c sample.c
worst_exit: 0
worst_mib: 4
headroom: 1/4
suggested_mib_per_job: 5
note: suggested_mib_per_job is the worst peak rounded up to MiB, plus headroom, rounded up; it is not a measurement of a named package

libxc’s row stays unmeasured until that launcher has been run on a libxc 7.0.0 build with those flags. Do not fill the number in before then.

Limits

  • Software with no entry in data/build-memory.toml is not checked. The count line shows how many pairs had a profile: fits plus oom.

  • A target with no node in the bot resource table gets a note and is not checked.

  • The model is linear in the parallel level. It does not model link steps, test steps that run with a different parallel level, or memory that other jobs on the node use. It does not model ulimit -u, which det_parallelism also consults.

  • --hooks reads the PARALLELISM_LIMITS literal. It does not run post_ready_hook; the rules that pick a row are the ones described under Where the numbers come from. Of the rest of the file it checks only for a statement that drops a sum. Other mistakes in the hooks are not found.

  • limit_by_memory and the memory cap do not replace the estimate. The estimate follows the hooks that are in eb_hooks.py. The --node report is the comparison, and the warnings for an easystack do not print the cap. A package with no measured mib_per_job stays unmeasured.

  • The MCP tool eb_stack_oom returns the same lines and each warning as JSON.