mod oom_profile¶
- module oom_profile¶
Bot jobs a plan entry is likely to run out of memory in.
data/build-memory.tomlrecords how much memory a build has needed, per EasyBuild name and version family, from EESSI bot jobs that ran out and jobs that did not.data/eessi-bot-resources.tomlrecords the cores and memory a bot job gets on each instance and CPU target, and thePARALLELISM_LIMITSthe EESSI hooks apply. Together they estimate the peak of each build in a plan on each target. An estimate that reaches the job’s memory is a warning, and the warning names the remedies the EESSI maintainers have used: a lower parallel level, or a node with more memory. Nothing is changed; the remedies are text.The hook ops are
divide_by_factorandset_maximumfromeb_hooks.py. A divide factor that is not an integer makes Python’s//return a float (16 // 2.5is6.0); the command warns and uses the numeric floor for the estimate.--hooksreads the rows from aneb_hooks.pyinstead of the table, throughcrate::eb_hooks.data/job-memory.tomlholdsmib_per_jobforlimit_by_memory. A row with no number is unmeasured.--nodeprints the hook parallelism beside that rule.Variables
- const BUNDLED_BOT_RESOURCES: &str¶
The bot job table shipped with eb-stack.
- const BUNDLED_BUILD_MEMORY: &str¶
The peak-memory table shipped with eb-stack.
- const BUNDLED_JOB_MEMORY: &str¶
The per-package
mib_per_jobtable shipped with eb-stack.
Functions
- fn divide_by_factor(parallel: u32, factor: DivideFactor) -> DividedParallel¶
divide_by_factorfromeb_hooks.py:max(1, parallel // factor).An integer factor uses integer division, so the result is an int. A float factor uses floor division and the result is a float when the quotient is at least 1.
levelis that number as a count of jobs.
- fn format_factor(factor: DivideFactor) -> String¶
Render a divide factor the way the table wrote it.
- fn job_memory_mib() -> Option<u64>¶
Memory for
limit_by_memoryon this process.The smallest cgroup v2
memory.maxon/proc/self/cgroup’s path, under/sys/fs/cgroup. A Slurm job’s limit is the one on that path. With no cgroup v2 limit, MemTotal, the same figure EasyBuild’sget_total_memory()reads.
- fn job_memory_mib_from(cgroup_text: &str, sysfs: &Path, meminfo: &str) -> Option<u64>¶
Memory for
limit_by_memory, from a cgroup listing, a cgroup filesystem, and a meminfo file.The smallest cgroup v2
memory.maxon the process path wins. With none, MemTotal.
- fn limit_by_memory(parallel: u32, mib_per_job: u32, memory_mib: u64) -> u32¶
limit_by_memory(parallel, mib_per_job)with the memory passed in.max(1, min(parallel, memory_mib // mib_per_job)).memory_mibis whatjob_memory_mibreads on the machine a build is running on, or the memory of a node size the report is asked about.mib_per_jobis the peak RSS of the worst compile. A zeromib_per_jobis not a budget; the table parser refuses it, and this returns 1.
- fn memory_cap(core_cap: u32, mib_per_job: u32, memory_mib: u64) -> u32¶
The core cap with a memory ceiling on top:
min(core_cap, memory_mib // mib_per_job).core_capis the level the hook leaves. Unlikelimit_by_memorythe result is not raised to 1: 0 says one job needs more memory than the node has. Flooring the MiB first does not change the quotient, so this is alsomemory_bytes // (mib_per_job * 2**20). A zeromib_per_jobis not a budget; the table parser refuses it, and this returnscore_cap.
- fn memtotal_mib(meminfo: &str) -> Option<u64>¶
MemTotalfrom/proc/meminfo, in MiB.The line is kilobytes. Dividing by 1024 matches EasyBuild’s
get_total_memory().
- fn parse_node_size(text: &str) -> Result<NodeSize>¶
Parse
CORES:MEMORY_MIBorCORES:MEMORY_MIB:TARGET.
- fn recipe_maxparallel(text: &str) -> Option<u32>¶
maxparallel = Nat the top level of an easyconfig.
- fn render_parallelism(resources: &BotResources, jobs: &JobMemoryTable, nodes: &[NodeSize]) -> String¶
Parallelism under the current hook and under
limit_by_memory.Each node is a core count, a memory in MiB, and an optional CPU target. The hook column is what
post_ready_hookwould leave, including EasyBuild’s--parallelandmax-parallel. The memory column islimit_by_memory(cores, mib_per_job)when the table has a number, andunmeasuredwhen it does not. With a number, the hook column is followed by itsmemory_cap. No number is invented.
- fn run_oom_check(request: &OomCheckRequest) -> Result<OomCheckReport>¶
Estimate each plan build’s peak on each bot node and warn where it reaches the job’s memory.
- fn set_maximum(parallel: u32, maximum: u32) -> u32¶
set_maximumfromeb_hooks.py:min(parallel, maximum).
- fn smallest_cgroup_memory_mib(cgroup_text: &str, sysfs: &Path) -> Option<u64>¶
Smallest cgroup v2
memory.maxoncgroup_text’s path, in MiB.sysfsis/sys/fs/cgroupon a host. The walk starts at the process cgroup and climbs tosysfs, and the smallest numericmemory.maxwins.maxis unlimited and does not count. No numeric file isNone, which is when the caller uses MemTotal.
Enums
- enum LimitOp¶
How a hook limit lowers parallelism.
- Divide(DivideFactor)¶
divide_by_factor:max(1, parallel // factor).
- Maximum(u32)¶
set_maximum:min(parallel, maximum).
Implementations
Structs and Unions
- struct BotResources¶
The bot job table and the EESSI hook limits.
- max_parallel: u32¶
EasyBuild’s default
max-parallel.
- parallel: Option<u32>¶
EasyBuild’s
--parallelfor the job.None, which is what the bot passes, lets EasyBuild count the cores.
- hooks_source: String¶
The
eb_hooks.pythe limits were copied from.
Implementations
- impl BotResources¶
Functions
- fn bundled() -> Result<Self>¶
The table shipped in
data/eessi-bot-resources.toml.
- fn factor_warnings(&self) -> Vec<String>¶
One line per divide factor that is not an integer literal.
eb_hooks.pycomputesmax(1, parallel // factor). A non-integer factor makes//return a float:16 // 2.5is6.0. A whole number written as a float does the same:16 // 2.0is8.0. A row read from aneb_hooks.pynames its file and line.
- fn hook_limit(&self, name: &str, target: &str) -> Option<LimitOp>¶
The hook limit for
nameontarget, aspost_ready_hookpicks it.
- fn load(path: &Path) -> Result<Self>¶
Read a table from
path.
- fn matching_limit(&self, name: &str, target: &str) -> Option<&HookLimit>¶
The hook row for
nameontarget, aspost_ready_hookpicks it.A software entry decides on its own: its row for the target, else its
*row, else none. A CPU-target key overrides*. Only software without an entry gets the target’s own limit, which on A64FX isdivide_by_factorby 4.
- fn parallel(&self, name: &str, partition: &Partition, recipe: Option<u32>) -> Parallel¶
The parallel level
namebuilds with onpartition.post_ready_hookapplies its op to the cores it detects, not to--parallelormax-parallel, and keeps the result only when it is below the level EasyBuild chose.
- fn parse(text: &str) -> Result<Self>¶
Parse the TOML document.
- struct DivideFactor¶
A
divide_by_factorargument.An integer literal (
divide = 2) makes Python’s//return an int. A float (divide = 2.5, ordivide = 2.0) makes//return a float:16 // 2.5is6.0and16 // 2.0is8.0.Implementations
- impl DivideFactor¶
Functions
- fn integer_literal(self) -> bool¶
The table wrote an integer,
divide = 2, not a float.
- fn value(self) -> f64¶
The factor as a number.
Traits implemented
- impl std::fmt::Debug for DivideFactor¶
- struct DividedParallel¶
What
divide_by_factorreturns, and whether Python’s result is a float.- level: u32¶
max(1, floor(parallel / factor)).
- float_result: bool¶
maxkept a float. True when the factor is a float and the quotient is at least 1. Below 1,max(1, 0.0)is the int 1.
- struct HookLimit¶
One
PARALLELISM_LIMITSrow.- software: Option<String>¶
EasyBuild name.
Noneis the target’s own limit.
- target: String¶
CPU target, or
*for every target.
- line: Option<usize>¶
Line of the
(op, argument)tuple in theeb_hooks.pythe row was read from.Nonefor a row of the bot resource table.
- struct JobMemory¶
One package’s
mib_per_job.Nonemeans the peak has not been measured.- name: String¶
EasyBuild name,
libxc.
- versions: Vec<String>¶
Version families,
7.0. Empty covers every version.
- mib_per_job: Option<u32>¶
MiB the worst compile needs, plus headroom. Absent when unmeasured.
- note: String¶
Why the number is there, or why it is not.
- struct JobMemoryTable¶
The per-package
mib_per_jobtable.Implementations
- impl JobMemoryTable¶
Functions
- fn bundled() -> Result<Self>¶
The table shipped in
data/job-memory.toml.
- fn load(path: &Path) -> Result<Self>¶
Read a table from
path.
- fn parse(text: &str) -> Result<Self>¶
Parse the TOML document. A package with no
mib_per_jobis unmeasured, and its note must say so. A number has to be positive.
- struct MemoryEntry¶
One recorded build memory figure.
- name: String¶
EasyBuild name,
libxc.
- versions: Vec<String>¶
Version families,
7.0. Empty covers every version.
- toolchains: Vec<String>¶
Toolchain families as the module names them,
GCC-15.2. Empty covers all.
- targets: Vec<String>¶
CPU targets or target prefixes,
x86_64/amd/zen4. Empty covers all.
- per_process_gib: f64¶
GiB each parallel build process needs.
- base_gib: f64¶
GiB the build needs whatever the parallel level.
- lower_bound: bool¶
The figures are the least the failures allow.
- reason: String¶
What uses the memory.
- remedy: Option<String>¶
What the maintainers did about it, when the thread says.
- sources: Vec<String>¶
Threads the figures come from.
- observations: Vec<Observation>¶
Bot jobs the figures have to reproduce.
Implementations
- impl MemoryEntry¶
Functions
- fn peak_gib(&self, parallel: u32) -> f64¶
Estimated peak GiB at
parallel.
- struct MemoryProfile¶
The peak-memory table.
Implementations
- impl MemoryProfile¶
Functions
- fn bundled() -> Result<Self>¶
The table shipped in
data/build-memory.toml.
- fn contradictions(&self) -> Vec<String>¶
Observations the table does not reproduce, one line each.
- fn entries(&self) -> &[MemoryEntry]¶
Every entry, in file order.
- fn load(path: &Path) -> Result<Self>¶
Read a table from
path.
- fn parse(text: &str) -> Result<Self>¶
Parse the TOML document. A table whose figures contradict one of its own observations is refused.
- struct NodeSize¶
One node size the parallelism report compares.
- cores: u32¶
CPUs the job holds.
- memory_mib: u64¶
Memory the job can use, in MiB.
- target: Option<String>¶
CPU target,
x86_64/amd/zen4. Absent applies only*hook rows.
- struct Observation¶
One bot job that ran out of memory, or did not.
- easyconfig: String¶
Easyconfig filename the job built.
- target: String¶
CPU target.
- parallel: Option<u32>¶
Parallel level the build ran with. Absent when the thread does not say, which only a build with no per-process figure allows.
- memory_gib: f64¶
GiB the job had.
- oom: bool¶
The job ran out of memory.
- source: String¶
The comment that reports it.
- struct OomCheckReport¶
What
stack oomprints.- stdout: String¶
Stdout, including the count line.
- warnings: Vec<OomWarning>¶
Builds likely to run out of memory.
- fits: usize¶
Build and node pairs with a memory figure that fit.
- builds: usize¶
Builds checked.
- partitions: usize¶
Node types checked.
- unknown_arches: Vec<String>¶
--archvalues the bot resource table has no node for.
- factor_warnings: Vec<String>¶
Divide factors that are not integers. Also included in
stdout.
- hook_warnings: Vec<String>¶
--hooksstatements that add to a name and drop the sum, such asmsg+ "...". Also included instdout.
- struct OomCheckRequest¶
Arguments for
stack oom.- arches: Vec<String>¶
CPU targets or target prefixes. Empty checks every target in the bot resource table.
- easystacks: Vec<PathBuf>¶
Easystacks whose entries are the plan. Repeatable.
- easyconfigs: Vec<PathBuf>¶
Robot trees the dependency closure is read from. Empty checks only the filenames the easystacks name.
- installed: Option<PathBuf>¶
Installed modules, one
Name/VersionFullper line. These are not built, so they are not checked.
- profile: Option<PathBuf>¶
Override
data/build-memory.toml.
- resources: Option<PathBuf>¶
Override
data/eessi-bot-resources.toml.
- nodes: Vec<String>¶
Node sizes,
CORES:MEMORY_MIBorCORES:MEMORY_MIB:TARGET. Prints the hook parallelism besidelimit_by_memory.
- job_memory: Option<PathBuf>¶
Override
data/job-memory.toml.
- hooks: Option<PathBuf>¶
An EESSI
eb_hooks.pywhosePARALLELISM_LIMITSreplace the hook rows of the bot resource table.
- parallel: Option<u32>¶
EasyBuild’s
--parallelfor the job.Nonecounts the cores.
- max_parallel: Option<u32>¶
EasyBuild’s
--max-parallelfor the job, in place of the table’s.
- struct OomWarning¶
One build likely to run out of memory in one bot job.
- file: String¶
Easyconfig filename.
- peak_gib: f64¶
Estimated peak GiB.
- entry: MemoryEntry¶
The memory table entry the estimate came from.
- required_by: BTreeSet<String>¶
Plan entries that need
file. Empty when the plan names it.
- remedies: Vec<String>¶
Remedies, one line each.
Implementations
- impl OomWarning¶
Functions
- fn lines(&self) -> Vec<String>¶
Every line the command prints for this warning.
- fn message(&self) -> String¶
The first line the command prints for this warning.
- struct Parallel¶
The parallel level a build gets in one job, and why.
- level: u32¶
The level EasyBuild runs with.
- cores: u32¶
CPUs the job holds.
- requested: Option<u32>¶
EasyBuild’s
--parallel, when the job sets one.
- max_parallel: Option<u32>¶
EasyBuild’s
max-parallel, when it lowered--parallelor the cores.
- hook: Option<LimitOp>¶
The hook limit that matched. It applies only when its result is below the level EasyBuild chose.
- recipe: Option<u32>¶
maxparallelin the easyconfig.
- struct Partition¶
One bot node type.
- instance: String¶
Bot instance,
eessi-bot-mc-aws.
- node: String¶
What the bot prints as “Building on”,
amd-zen4.
- target: String¶
What the bot prints as “Building for”,
x86_64/amd/zen4.
- accel: Option<String>¶
Accelerator the node carries,
nvidia/cc90.
- cores: u32¶
CPUs a job holds.
- memory_gib: f64¶
GiB a build in the job can use.
- sources: Vec<String>¶
Where the figures come from.
Implementations