mod eb_run

module eb_run

Run EasyBuild with easyblocks from the workspace, and read what it wrote.

An easyblock change is only proven by EasyBuild running it: the commands the easyblock generates (eb -x, which compiles nothing) and the build and test steps of a real build. These helpers start eb with --include-easyblocks on workspace files and turn its output into the parts an agent needs: the commands per step, the test step’s output, and the error lines. Real builds are refused unless EB_STACK_ALLOW_BUILD=1, so a server on a machine that must not build never starts one.

Variables

const ALLOW_BUILD_ENV: &str

Environment variable that must be 1 before run_eb starts a build.

const SHOW_CONFIG_OUTPUT_LIMIT: u64

Most output an eb --show-config probe may write to one file. The report is a few kilobytes; output near this size is not that report.

Functions

fn build_test_description(allowed: bool) -> String

The eb_build_test description on a server where builds are allowed or not, so the tool list itself says whether a build can run here.

fn builds_allowed() -> bool

True when this process may start a real build.

fn commands_in(text: &str) -> Vec<String>

Commands EasyBuild reports running, in order.

fn config_log_failure(text: &str) -> Option<ConfigLogFailure>

The last check in an autoconf config.log whose test command failed ($? = <non-zero>` or failed program was:`), before the cache variables. ``None when no check failed.

fn configured_robot_paths(program: &Path, output_file: &Path, deadline: Deadline, extra_env: &[(String, String)]) -> Result<Vec<String>, String>

The robot paths program is configured with, in EasyBuild’s order, as eb --show-config reports robot-paths: from EASYBUILD_ROBOT_PATHS, a configuration file, or the built-in default, with EASYBUILD_ROBOT paths first. EasyBuild reads its own configuration, so every source and template it supports is honoured.

The probe runs as run_eb_with runs a dry run, in the directory of output_file with extra_env, so it sees the run’s configuration. It stops at deadline, and when a file it writes reaches SHOW_CONFIG_OUTPUT_LIMIT; then it is stopped with every process it started. Its output goes to output_file, read back at most that size.

fn dry_run_environment(text: &str, filter: Option<&str>) -> std::collections::BTreeMap<String, String>

The build environment an eb -x run reports (export NAME="value" and setting environment variable NAME to "value" lines): what the toolchain and the loaded dependencies define, in the form the easyblock reads it. With filter, only names containing it (case-insensitive).

fn dry_run_guard(dir: &Path) -> Result<Vec<(String, String)>, String>

sitecustomize so OpenMPI.cleanup does not remove the eb -x session directory.

fn dry_run_next(allowed: bool, finished: bool) -> &'static str

The next field of a dry run: what to do after it, given whether builds are allowed and whether the dry run finished.

fn error_lines(text: &str, limit: usize) -> Vec<String>

Lines of text that look like errors, deduplicated, at most limit. An Lmod error comes first, as one line with its wrapped message joined.

fn failed_dry_run_text(kind: Option<&str>, errors: &[String]) -> (String, &'static str)

The summary and next of a failed dry run of the given kind, quoting the first error line when there is one.

fn failure_kind(errors: &[String]) -> Option<&'static str>

Where a failed eb run went wrong, from its error lines: setup when EasyBuild itself never started because a module failed to load (Lmod printed an error); environment when EasyBuild could not set up the recipe’s toolchain or dependencies here, so no easyblock code ran; easyblock when Python failed in a module. None when the lines say none of these.

fn find_config_log(dir: &Path) -> Option<PathBuf>

The most recently written config.log under dir, at most six levels down, without following symlinks.

fn last_step_in_file(path: &Path) -> Option<(String, usize)>

The last step a log starts and its 1-based line, read line by line over the whole file. For a log whose tail holds no step marker because one step’s output alone is longer than the tail.

fn lmod_error(text: &str) -> Option<String>

The first Lmod error in text as one line, its wrapped message joined: Lmod ends the message at While processing the following module(s): or a blank line.

fn parse_dry_run(text: &str) -> Vec<DryRunStep>

Group the commands of an eb -x run by step.

fn parse_show_config_robot_paths(text: &str) -> Result<Vec<String>, String>

The robot-paths value of eb --show-config output: robot-paths (E) = /a, /b, with paths separated by a comma and a space.

fn read_tail(path: &Path, max_bytes: u64) -> Result<(String, bool), String>

Read the end of a file as text, at most max_bytes from its end, so a large build log stays bounded.

fn recipe_toolchain_options(text: &str) -> std::collections::BTreeMap<String, String>

The toolchainopts of an easyconfig as written (toolchainopts = {'pic': True, 'openmp': True}): option name to the text of its value. Empty when the recipe sets none. Commented entries are skipped.

fn resolve_eb() -> Result<PathBuf, String>

The eb program: EB_STACK_EB when set, else eb on PATH.

fn run_eb(args: &[String], limit: Duration, output_file: &Path) -> Result<EbRun, String>

Run eb args..., stdout and stderr both to output_file, killed after limit. The caller decides whether the run may build.

eb starts in the directory of output_file: python -m easybuild.main puts the current directory first on sys.path, so starting it at the root of an easyblocks checkout would import that checkout’s easybuild package instead of the installed framework.

fn run_eb_with(args: &[String], limit: Duration, output_file: &Path, extra_env: &[(String, String)]) -> Result<EbRun, String>

run_eb, with extra environment for the child process.

fn run_program(program: &Path, args: &[String], limit: Duration, output_file: &Path) -> Result<EbRun, String>

run_eb with the program given. At the limit, every process the program started goes with it (stop_tree).

fn split_steps(text: &str) -> Vec<LogStep>

Split a log into steps. A step runs until the next step starts; lines before the first step belong to start. A step that starts twice (the announcement and the log line) is one span.

fn split_warnings(lines: Vec<String>) -> (Vec<String>, Vec<String>)

Split error lines into real errors and the warnings EasyBuild printed about errors it ignored (!!! WARNING: ignoring error ...), which do not stop a run.

fn step_started(line: &str) -> Option<String>

The step a line starts, when it starts one.

fn step_text<'a>(text: &'a str, steps: &[LogStep], step: &str) -> Option<Vec<&'a str>>

The text of one step (the last span with that name), or None.

fn tail(lines: &[&str], n: usize) -> String

The last n of lines, joined.

fn threaded_findings(options: &std::collections::BTreeMap<String, String>, pairs: &[ThreadedPair], steps: &[DryRunStep]) -> Vec<ThreadedFinding>

Generated commands that use a plain library value where the recipe’s threaded toolchain option asks for the threaded one. A command matches a pair when it has every library of the plain value and none of the libraries the threaded value adds. Pairs with the same missing libraries on the same command make one finding. Empty when no threaded option is on.

fn threaded_findings_text(options: &std::collections::BTreeMap<String, String>, findings: &[ThreadedFinding]) -> Option<(String, String)>

The summary and next of a finished dry run with threaded findings: what is wrong in which command, and the edit that fixes it.

fn threaded_note(options: &std::collections::BTreeMap<String, String>, pairs: &[ThreadedPair]) -> Option<String>

What a dry run should tell the agent about threaded variants: Some when the recipe turns a threaded toolchain option on and the environment has plain/threaded pairs, since the commands then look the same as without the option unless the easyblock picks the variant itself.

fn threaded_options_on(options: &std::collections::BTreeMap<String, String>) -> Vec<&'static str>

The threaded toolchain options a recipe turns on.

fn threaded_pairs(env: &std::collections::BTreeMap<String, String>) -> Vec<ThreadedPair>

The plain/threaded pairs of a build environment. EasyBuild’s toolchains define a threaded variant next to several library variables, with an MT part in the name; which one applies is up to the easyblock, so a dry run of a recipe with a threaded toolchain option reports both the same way.

Structs and Unions

struct ConfigLogFailure

The check an autoconf configure failed on, from its config.log.

check: String

The checking ... text of the check.

line: usize

1-based line of the check in config.log.

excerpt: String

The check’s commands and compiler or linker output, at most 40 lines (the test program’s source is left out), and the error: line.

struct Deadline

One time limit shared by the steps of an operation, such as the eb --show-config probe and the run that follows it.

Implementations

impl Deadline

Functions

fn after(limit: Duration) -> Self

A deadline limit from now.

fn limit(&self) -> Duration

The whole limit the deadline was set with.

fn remaining(&self) -> Duration

Time left before the deadline; zero once it has passed.

struct DryRunStep

Commands per step from eb -x output.

step: String

Step name.

commands: Vec<String>

Commands the easyblock would run in it.

struct EbInvocation

What to build: a recipe, the easyblock modules to include, extra robot paths and --try-amend values.

recipe: PathBuf

The easyconfig.

easyblocks: Vec<PathBuf>

Easyblock modules for --include-easyblocks.

robot: Vec<PathBuf>

Robot paths searched before the ones EasyBuild is configured with.

try_amend: Vec<String>

key=value pairs for --try-amend.

Implementations

impl EbInvocation

Functions

fn args(&self, extra: &[String]) -> Vec<String>

eb arguments for this invocation followed by extra, for an EasyBuild with no robot paths configured: the trailing : in --robot-paths stands for EasyBuild’s built-in default. Use EbInvocation::args_with_configured where EASYBUILD_ROBOT_PATHS or a configuration file may set them.

fn args_with_configured(&self, configured: &[String], extra: &[String]) -> Result<Vec<String>, String>

EbInvocation::args, with robot searched before configured, the robot paths EasyBuild is configured with (configured_robot_paths).

--robot-paths replaces the configured value, and an empty element in it stands for EasyBuild’s built-in default only, not for a value set in EASYBUILD_ROBOT_PATHS or a configuration file. So the configured paths are spelled out after robot. Without robot the arguments carry no --robot-paths and EasyBuild keeps its own.

fn dry_run_args(&self, dir: &Path) -> Vec<String>

eb -x arguments for this invocation, with the temporary files kept in <dir>/tmp, as EbInvocation::args gives a real build. EasyBuild 5.4.1 can exit 1 after a dry run that succeeded, while it removes a temporary log that is already gone (ERROR: Failed to remove log file(s) … [Errno 2]` after Build succeeded for 1 out of 1`), and a dry run that exits 1 has its findings dropped. Builds already pass ``--disable-cleanup-tmpdir; dry runs pass it too. OpenMPI’s own removal of $TMPDIR is refused by dry_run_guard, not by this flag.

fn dry_run_args_with_configured(&self, dir: &Path, configured: &[String]) -> Result<Vec<String>, String>

EbInvocation::dry_run_args, with robot searched before configured, the robot paths EasyBuild is configured with (configured_robot_paths).

fn resolved(&self, root: &Path) -> Result<EbInvocation, String>

Check the invocation against the workspace: the recipe and every easyblock resolve inside root, easyblocks are .py files, and each --try-amend value is name=value with a lowercase parameter name.

struct EbRun

How one eb run ended.

argv: Vec<String>

The command line, program first.

exit_code: Option<i32>

Exit code; None when killed.

timed_out: bool

True when the run was killed at its time limit.

seconds: u64

Wall time in seconds.

output_file: PathBuf

Where stdout and stderr went.

struct LogStep

One step’s span in a log: first and end are 0-based line indexes, end exclusive.

step: String

Step name (configure, test, …).

first_line: usize

First line of the step, 1-based.

lines: usize

Number of lines in the step.

struct ThreadedFinding

A generated command that carries the value of a plain library variable while the recipe turns a threaded toolchain option on, and lacks what the threaded variant of that variable adds.

step: String

The step the command runs in.

command: String

The command, at most 300 characters.

plain: Vec<String>

Plain variables whose value the command carries.

threaded: Vec<String>

The threaded variable of each entry of plain, in the same order.

missing: Vec<String>

Libraries the threaded variants add that the command does not have.

struct ThreadedPair

A build-environment variable and its threaded variant, when their values differ.

plain: String

The plain name (LIBX, X_STATIC_LIBS).

threaded: String

The threaded name (LIBX_MT, X_MT_STATIC_LIBS, X_STATIC_LIBS_MT).

plain_value: String

Value of the plain name.

threaded_value: String

Value of the threaded name.