mod oom_profile

module oom_profile

Bot jobs a plan entry is likely to run out of memory in.

data/build-memory.toml records how much memory a build has needed, per EasyBuild name and version family, from EESSI bot jobs that ran out and jobs that did not. data/eessi-bot-resources.toml records the cores and memory a bot job gets on each instance and CPU target, and the PARALLELISM_LIMITS the EESSI hooks apply. Together they estimate the peak of each build in a plan on each target. An estimate that reaches the job’s memory is a warning, and the warning names the remedies the EESSI maintainers have used: a lower parallel level, or a node with more memory. Nothing is changed; the remedies are text.

The hook ops are divide_by_factor and set_maximum from eb_hooks.py. A divide factor that is not an integer makes Python’s // return a float (16 // 2.5 is 6.0); the command warns and uses the numeric floor for the estimate. --hooks reads the rows from an eb_hooks.py instead of the table, through crate::eb_hooks. data/job-memory.toml holds mib_per_job for limit_by_memory. A row with no number is unmeasured. --node prints the hook parallelism beside that rule.

Variables

const BUNDLED_BOT_RESOURCES: &str

The bot job table shipped with eb-stack.

const BUNDLED_BUILD_MEMORY: &str

The peak-memory table shipped with eb-stack.

const BUNDLED_JOB_MEMORY: &str

The per-package mib_per_job table shipped with eb-stack.

Functions

fn divide_by_factor(parallel: u32, factor: DivideFactor) -> DividedParallel

divide_by_factor from eb_hooks.py: max(1, parallel // factor).

An integer factor uses integer division, so the result is an int. A float factor uses floor division and the result is a float when the quotient is at least 1. level is that number as a count of jobs.

fn format_factor(factor: DivideFactor) -> String

Render a divide factor the way the table wrote it.

fn job_memory_mib() -> Option<u64>

Memory for limit_by_memory on this process.

The smallest cgroup v2 memory.max on /proc/self/cgroup’s path, under /sys/fs/cgroup. A Slurm job’s limit is the one on that path. With no cgroup v2 limit, MemTotal, the same figure EasyBuild’s get_total_memory() reads.

fn job_memory_mib_from(cgroup_text: &str, sysfs: &Path, meminfo: &str) -> Option<u64>

Memory for limit_by_memory, from a cgroup listing, a cgroup filesystem, and a meminfo file.

The smallest cgroup v2 memory.max on the process path wins. With none, MemTotal.

fn limit_by_memory(parallel: u32, mib_per_job: u32, memory_mib: u64) -> u32

limit_by_memory(parallel, mib_per_job) with the memory passed in.

max(1, min(parallel, memory_mib // mib_per_job)). memory_mib is what job_memory_mib reads on the machine a build is running on, or the memory of a node size the report is asked about. mib_per_job is the peak RSS of the worst compile. A zero mib_per_job is not a budget; the table parser refuses it, and this returns 1.

fn memory_cap(core_cap: u32, mib_per_job: u32, memory_mib: u64) -> u32

The core cap with a memory ceiling on top: min(core_cap, memory_mib // mib_per_job).

core_cap is the level the hook leaves. Unlike limit_by_memory the result is not raised to 1: 0 says one job needs more memory than the node has. Flooring the MiB first does not change the quotient, so this is also memory_bytes // (mib_per_job * 2**20). A zero mib_per_job is not a budget; the table parser refuses it, and this returns core_cap.

fn memtotal_mib(meminfo: &str) -> Option<u64>

MemTotal from /proc/meminfo, in MiB.

The line is kilobytes. Dividing by 1024 matches EasyBuild’s get_total_memory().

fn parse_node_size(text: &str) -> Result<NodeSize>

Parse CORES:MEMORY_MIB or CORES:MEMORY_MIB:TARGET.

fn recipe_maxparallel(text: &str) -> Option<u32>

maxparallel = N at the top level of an easyconfig.

fn render_parallelism(resources: &BotResources, jobs: &JobMemoryTable, nodes: &[NodeSize]) -> String

Parallelism under the current hook and under limit_by_memory.

Each node is a core count, a memory in MiB, and an optional CPU target. The hook column is what post_ready_hook would leave, including EasyBuild’s --parallel and max-parallel. The memory column is limit_by_memory(cores, mib_per_job) when the table has a number, and unmeasured when it does not. With a number, the hook column is followed by its memory_cap. No number is invented.

fn run_oom_check(request: &OomCheckRequest) -> Result<OomCheckReport>

Estimate each plan build’s peak on each bot node and warn where it reaches the job’s memory.

fn set_maximum(parallel: u32, maximum: u32) -> u32

set_maximum from eb_hooks.py: min(parallel, maximum).

fn smallest_cgroup_memory_mib(cgroup_text: &str, sysfs: &Path) -> Option<u64>

Smallest cgroup v2 memory.max on cgroup_text’s path, in MiB.

sysfs is /sys/fs/cgroup on a host. The walk starts at the process cgroup and climbs to sysfs, and the smallest numeric memory.max wins. max is unlimited and does not count. No numeric file is None, which is when the caller uses MemTotal.

Enums

enum LimitOp

How a hook limit lowers parallelism.

Divide(DivideFactor)

divide_by_factor: max(1, parallel // factor).

Maximum(u32)

set_maximum: min(parallel, maximum).

Implementations

impl LimitOp

Structs and Unions

struct BotResources

The bot job table and the EESSI hook limits.

max_parallel: u32

EasyBuild’s default max-parallel.

parallel: Option<u32>

EasyBuild’s --parallel for the job. None, which is what the bot passes, lets EasyBuild count the cores.

partitions: Vec<Partition>

Node types, in file order.

limits: Vec<HookLimit>

Hook limits, in file order.

hooks_source: String

The eb_hooks.py the limits were copied from.

Implementations

impl BotResources

Functions

fn bundled() -> Result<Self>

The table shipped in data/eessi-bot-resources.toml.

fn factor_warnings(&self) -> Vec<String>

One line per divide factor that is not an integer literal.

eb_hooks.py computes max(1, parallel // factor). A non-integer factor makes // return a float: 16 // 2.5 is 6.0. A whole number written as a float does the same: 16 // 2.0 is 8.0. A row read from an eb_hooks.py names its file and line.

fn hook_limit(&self, name: &str, target: &str) -> Option<LimitOp>

The hook limit for name on target, as post_ready_hook picks it.

fn load(path: &Path) -> Result<Self>

Read a table from path.

fn matching_limit(&self, name: &str, target: &str) -> Option<&HookLimit>

The hook row for name on target, as post_ready_hook picks it.

A software entry decides on its own: its row for the target, else its * row, else none. A CPU-target key overrides *. Only software without an entry gets the target’s own limit, which on A64FX is divide_by_factor by 4.

fn parallel(&self, name: &str, partition: &Partition, recipe: Option<u32>) -> Parallel

The parallel level name builds with on partition.

post_ready_hook applies its op to the cores it detects, not to --parallel or max-parallel, and keeps the result only when it is below the level EasyBuild chose.

fn parse(text: &str) -> Result<Self>

Parse the TOML document.

struct DivideFactor

A divide_by_factor argument.

An integer literal (divide = 2) makes Python’s // return an int. A float (divide = 2.5, or divide = 2.0) makes // return a float: 16 // 2.5 is 6.0 and 16 // 2.0 is 8.0.

Implementations

impl DivideFactor

Functions

fn integer_literal(self) -> bool

The table wrote an integer, divide = 2, not a float.

fn value(self) -> f64

The factor as a number.

Traits implemented

impl std::fmt::Debug for DivideFactor
struct DividedParallel

What divide_by_factor returns, and whether Python’s result is a float.

level: u32

max(1, floor(parallel / factor)).

float_result: bool

max kept a float. True when the factor is a float and the quotient is at least 1. Below 1, max(1, 0.0) is the int 1.

struct HookLimit

One PARALLELISM_LIMITS row.

software: Option<String>

EasyBuild name. None is the target’s own limit.

target: String

CPU target, or * for every target.

op: LimitOp

The limit.

line: Option<usize>

Line of the (op, argument) tuple in the eb_hooks.py the row was read from. None for a row of the bot resource table.

struct JobMemory

One package’s mib_per_job. None means the peak has not been measured.

name: String

EasyBuild name, libxc.

versions: Vec<String>

Version families, 7.0. Empty covers every version.

mib_per_job: Option<u32>

MiB the worst compile needs, plus headroom. Absent when unmeasured.

note: String

Why the number is there, or why it is not.

struct JobMemoryTable

The per-package mib_per_job table.

Implementations

impl JobMemoryTable

Functions

fn bundled() -> Result<Self>

The table shipped in data/job-memory.toml.

fn get(&self, name: &str) -> Option<&JobMemory>

The row for name, if the table has one.

fn load(path: &Path) -> Result<Self>

Read a table from path.

fn packages(&self) -> &[JobMemory]

Every package, in file order.

fn parse(text: &str) -> Result<Self>

Parse the TOML document. A package with no mib_per_job is unmeasured, and its note must say so. A number has to be positive.

struct MemoryEntry

One recorded build memory figure.

name: String

EasyBuild name, libxc.

versions: Vec<String>

Version families, 7.0. Empty covers every version.

toolchains: Vec<String>

Toolchain families as the module names them, GCC-15.2. Empty covers all.

targets: Vec<String>

CPU targets or target prefixes, x86_64/amd/zen4. Empty covers all.

per_process_gib: f64

GiB each parallel build process needs.

base_gib: f64

GiB the build needs whatever the parallel level.

lower_bound: bool

The figures are the least the failures allow.

reason: String

What uses the memory.

remedy: Option<String>

What the maintainers did about it, when the thread says.

sources: Vec<String>

Threads the figures come from.

observations: Vec<Observation>

Bot jobs the figures have to reproduce.

Implementations

impl MemoryEntry

Functions

fn peak_gib(&self, parallel: u32) -> f64

Estimated peak GiB at parallel.

struct MemoryProfile

The peak-memory table.

Implementations

impl MemoryProfile

Functions

fn bundled() -> Result<Self>

The table shipped in data/build-memory.toml.

fn contradictions(&self) -> Vec<String>

Observations the table does not reproduce, one line each.

fn entries(&self) -> &[MemoryEntry]

Every entry, in file order.

fn load(path: &Path) -> Result<Self>

Read a table from path.

fn parse(text: &str) -> Result<Self>

Parse the TOML document. A table whose figures contradict one of its own observations is refused.

struct NodeSize

One node size the parallelism report compares.

cores: u32

CPUs the job holds.

memory_mib: u64

Memory the job can use, in MiB.

target: Option<String>

CPU target, x86_64/amd/zen4. Absent applies only * hook rows.

struct Observation

One bot job that ran out of memory, or did not.

easyconfig: String

Easyconfig filename the job built.

target: String

CPU target.

parallel: Option<u32>

Parallel level the build ran with. Absent when the thread does not say, which only a build with no per-process figure allows.

memory_gib: f64

GiB the job had.

oom: bool

The job ran out of memory.

source: String

The comment that reports it.

struct OomCheckReport

What stack oom prints.

stdout: String

Stdout, including the count line.

warnings: Vec<OomWarning>

Builds likely to run out of memory.

fits: usize

Build and node pairs with a memory figure that fit.

builds: usize

Builds checked.

partitions: usize

Node types checked.

unknown_arches: Vec<String>

--arch values the bot resource table has no node for.

factor_warnings: Vec<String>

Divide factors that are not integers. Also included in stdout.

hook_warnings: Vec<String>

--hooks statements that add to a name and drop the sum, such as msg+ "...". Also included in stdout.

struct OomCheckRequest

Arguments for stack oom.

arches: Vec<String>

CPU targets or target prefixes. Empty checks every target in the bot resource table.

easystacks: Vec<PathBuf>

Easystacks whose entries are the plan. Repeatable.

easyconfigs: Vec<PathBuf>

Robot trees the dependency closure is read from. Empty checks only the filenames the easystacks name.

installed: Option<PathBuf>

Installed modules, one Name/VersionFull per line. These are not built, so they are not checked.

profile: Option<PathBuf>

Override data/build-memory.toml.

resources: Option<PathBuf>

Override data/eessi-bot-resources.toml.

nodes: Vec<String>

Node sizes, CORES:MEMORY_MIB or CORES:MEMORY_MIB:TARGET. Prints the hook parallelism beside limit_by_memory.

job_memory: Option<PathBuf>

Override data/job-memory.toml.

hooks: Option<PathBuf>

An EESSI eb_hooks.py whose PARALLELISM_LIMITS replace the hook rows of the bot resource table.

parallel: Option<u32>

EasyBuild’s --parallel for the job. None counts the cores.

max_parallel: Option<u32>

EasyBuild’s --max-parallel for the job, in place of the table’s.

struct OomWarning

One build likely to run out of memory in one bot job.

file: String

Easyconfig filename.

partition: Partition

The bot node type.

parallel: Parallel

The parallel level the build would get.

peak_gib: f64

Estimated peak GiB.

entry: MemoryEntry

The memory table entry the estimate came from.

required_by: BTreeSet<String>

Plan entries that need file. Empty when the plan names it.

remedies: Vec<String>

Remedies, one line each.

Implementations

impl OomWarning

Functions

fn lines(&self) -> Vec<String>

Every line the command prints for this warning.

fn message(&self) -> String

The first line the command prints for this warning.

struct Parallel

The parallel level a build gets in one job, and why.

level: u32

The level EasyBuild runs with.

cores: u32

CPUs the job holds.

requested: Option<u32>

EasyBuild’s --parallel, when the job sets one.

max_parallel: Option<u32>

EasyBuild’s max-parallel, when it lowered --parallel or the cores.

hook: Option<LimitOp>

The hook limit that matched. It applies only when its result is below the level EasyBuild chose.

recipe: Option<u32>

maxparallel in the easyconfig.

struct Partition

One bot node type.

instance: String

Bot instance, eessi-bot-mc-aws.

node: String

What the bot prints as “Building on”, amd-zen4.

target: String

What the bot prints as “Building for”, x86_64/amd/zen4.

accel: Option<String>

Accelerator the node carries, nvidia/cc90.

cores: u32

CPUs a job holds.

memory_gib: f64

GiB a build in the job can use.

sources: Vec<String>

Where the figures come from.

Implementations

impl Partition