mod eb_run¶
- module eb_run¶
Run EasyBuild with easyblocks from the workspace, and read what it wrote.
An easyblock change is only proven by EasyBuild running it: the commands the easyblock generates (
eb -x, which compiles nothing) and the build and test steps of a real build. These helpers startebwith--include-easyblockson workspace files and turn its output into the parts an agent needs: the commands per step, the test step’s output, and the error lines. Real builds are refused unlessEB_STACK_ALLOW_BUILD=1, so a server on a machine that must not build never starts one.Variables
- const ALLOW_BUILD_ENV: &str¶
Environment variable that must be
1beforerun_ebstarts a build.
- const SHOW_CONFIG_OUTPUT_LIMIT: u64¶
Most output an
eb --show-configprobe may write to one file. The report is a few kilobytes; output near this size is not that report.
Functions
- fn build_test_description(allowed: bool) -> String¶
The
eb_build_testdescription on a server where builds areallowedor not, so the tool list itself says whether a build can run here.
- fn builds_allowed() -> bool¶
True when this process may start a real build.
- fn commands_in(text: &str) -> Vec<String>¶
Commands EasyBuild reports running, in order.
- fn config_log_failure(text: &str) -> Option<ConfigLogFailure>¶
The last check in an autoconf
config.logwhose test command failed ($? = <non-zero>` orfailed program was:`), before the cache variables. ``Nonewhen no check failed.
- fn configured_robot_paths(program: &Path, output_file: &Path, deadline: Deadline, extra_env: &[(String, String)]) -> Result<Vec<String>, String>¶
The robot paths
programis configured with, in EasyBuild’s order, aseb --show-configreportsrobot-paths: fromEASYBUILD_ROBOT_PATHS, a configuration file, or the built-in default, withEASYBUILD_ROBOTpaths first. EasyBuild reads its own configuration, so every source and template it supports is honoured.The probe runs as
run_eb_withruns a dry run, in the directory ofoutput_filewithextra_env, so it sees the run’s configuration. It stops atdeadline, and when a file it writes reachesSHOW_CONFIG_OUTPUT_LIMIT; then it is stopped with every process it started. Its output goes tooutput_file, read back at most that size.
- fn dry_run_environment(text: &str, filter: Option<&str>) -> std::collections::BTreeMap<String, String>¶
The build environment an
eb -xrun reports (export NAME="value"andsetting environment variable NAME to "value"lines): what the toolchain and the loaded dependencies define, in the form the easyblock reads it. Withfilter, only names containing it (case-insensitive).
- fn dry_run_guard(dir: &Path) -> Result<Vec<(String, String)>, String>¶
sitecustomizeso OpenMPI.cleanup does not remove theeb -xsession directory.
- fn dry_run_next(allowed: bool, finished: bool) -> &'static str¶
The
nextfield of a dry run: what to do after it, given whether builds areallowedand whether the dry runfinished.
- fn error_lines(text: &str, limit: usize) -> Vec<String>¶
Lines of
textthat look like errors, deduplicated, at mostlimit. An Lmod error comes first, as one line with its wrapped message joined.
- fn failed_dry_run_text(kind: Option<&str>, errors: &[String]) -> (String, &'static str)¶
The summary and
nextof a failed dry run of the givenkind, quoting the first error line when there is one.
- fn failure_kind(errors: &[String]) -> Option<&'static str>¶
Where a failed
ebrun went wrong, from its error lines:setupwhen EasyBuild itself never started because a module failed to load (Lmod printed an error);environmentwhen EasyBuild could not set up the recipe’s toolchain or dependencies here, so no easyblock code ran;easyblockwhen Python failed in a module.Nonewhen the lines say none of these.
- fn find_config_log(dir: &Path) -> Option<PathBuf>¶
The most recently written
config.logunderdir, at most six levels down, without following symlinks.
- fn last_step_in_file(path: &Path) -> Option<(String, usize)>¶
The last step a log starts and its 1-based line, read line by line over the whole file. For a log whose tail holds no step marker because one step’s output alone is longer than the tail.
- fn lmod_error(text: &str) -> Option<String>¶
The first Lmod error in
textas one line, its wrapped message joined: Lmod ends the message atWhile processing the following module(s):or a blank line.
- fn parse_dry_run(text: &str) -> Vec<DryRunStep>¶
Group the commands of an
eb -xrun by step.
- fn parse_show_config_robot_paths(text: &str) -> Result<Vec<String>, String>¶
The
robot-pathsvalue ofeb --show-configoutput:robot-paths (E) = /a, /b, with paths separated by a comma and a space.
- fn read_tail(path: &Path, max_bytes: u64) -> Result<(String, bool), String>¶
Read the end of a file as text, at most
max_bytesfrom its end, so a large build log stays bounded.
- fn recipe_toolchain_options(text: &str) -> std::collections::BTreeMap<String, String>¶
The
toolchainoptsof an easyconfig as written (toolchainopts = {'pic': True, 'openmp': True}): option name to the text of its value. Empty when the recipe sets none. Commented entries are skipped.
- fn resolve_eb() -> Result<PathBuf, String>¶
The
ebprogram:EB_STACK_EBwhen set, elseebonPATH.
- fn run_eb(args: &[String], limit: Duration, output_file: &Path) -> Result<EbRun, String>¶
Run
eb args..., stdout and stderr both tooutput_file, killed afterlimit. The caller decides whether the run may build.ebstarts in the directory ofoutput_file:python -m easybuild.mainputs the current directory first onsys.path, so starting it at the root of an easyblocks checkout would import that checkout’seasybuildpackage instead of the installed framework.
- fn run_eb_with(args: &[String], limit: Duration, output_file: &Path, extra_env: &[(String, String)]) -> Result<EbRun, String>¶
run_eb, with extra environment for the child process.
- fn run_program(program: &Path, args: &[String], limit: Duration, output_file: &Path) -> Result<EbRun, String>¶
run_ebwith the program given. At the limit, every process the program started goes with it (stop_tree).
- fn split_steps(text: &str) -> Vec<LogStep>¶
Split a log into steps. A step runs until the next step starts; lines before the first step belong to
start. A step that starts twice (the announcement and the log line) is one span.
- fn split_warnings(lines: Vec<String>) -> (Vec<String>, Vec<String>)¶
Split error lines into real errors and the warnings EasyBuild printed about errors it ignored (
!!! WARNING: ignoring error ...), which do not stop a run.
- fn step_started(line: &str) -> Option<String>¶
The step a line starts, when it starts one.
- fn step_text<'a>(text: &'a str, steps: &[LogStep], step: &str) -> Option<Vec<&'a str>>¶
The text of one step (the last span with that name), or
None.
- fn tail(lines: &[&str], n: usize) -> String¶
The last
noflines, joined.
- fn threaded_findings(options: &std::collections::BTreeMap<String, String>, pairs: &[ThreadedPair], steps: &[DryRunStep]) -> Vec<ThreadedFinding>¶
Generated commands that use a plain library value where the recipe’s threaded toolchain option asks for the threaded one. A command matches a pair when it has every library of the plain value and none of the libraries the threaded value adds. Pairs with the same missing libraries on the same command make one finding. Empty when no threaded option is on.
- fn threaded_findings_text(options: &std::collections::BTreeMap<String, String>, findings: &[ThreadedFinding]) -> Option<(String, String)>¶
The
summaryandnextof a finished dry run with threaded findings: what is wrong in which command, and the edit that fixes it.
- fn threaded_note(options: &std::collections::BTreeMap<String, String>, pairs: &[ThreadedPair]) -> Option<String>¶
What a dry run should tell the agent about threaded variants:
Somewhen the recipe turns a threaded toolchain option on and the environment has plain/threaded pairs, since the commands then look the same as without the option unless the easyblock picks the variant itself.
- fn threaded_options_on(options: &std::collections::BTreeMap<String, String>) -> Vec<&'static str>¶
The threaded toolchain options a recipe turns on.
- fn threaded_pairs(env: &std::collections::BTreeMap<String, String>) -> Vec<ThreadedPair>¶
The plain/threaded pairs of a build environment. EasyBuild’s toolchains define a threaded variant next to several library variables, with an
MTpart in the name; which one applies is up to the easyblock, so a dry run of a recipe with a threaded toolchain option reports both the same way.
Structs and Unions
- struct ConfigLogFailure¶
The check an autoconf
configurefailed on, from itsconfig.log.- check: String¶
The
checking ...text of the check.
- line: usize¶
1-based line of the check in config.log.
- excerpt: String¶
The check’s commands and compiler or linker output, at most 40 lines (the test program’s source is left out), and the
error:line.
- struct Deadline¶
One time limit shared by the steps of an operation, such as the
eb --show-configprobe and the run that follows it.Implementations
- struct DryRunStep¶
Commands per step from
eb -xoutput.- step: String¶
Step name.
- commands: Vec<String>¶
Commands the easyblock would run in it.
- struct EbInvocation¶
What to build: a recipe, the easyblock modules to include, extra robot paths and
--try-amendvalues.- recipe: PathBuf¶
The easyconfig.
- easyblocks: Vec<PathBuf>¶
Easyblock modules for
--include-easyblocks.
- robot: Vec<PathBuf>¶
Robot paths searched before the ones EasyBuild is configured with.
- try_amend: Vec<String>¶
key=valuepairs for--try-amend.
Implementations
- impl EbInvocation¶
Functions
- fn args(&self, extra: &[String]) -> Vec<String>¶
ebarguments for this invocation followed byextra, for an EasyBuild with no robot paths configured: the trailing:in--robot-pathsstands for EasyBuild’s built-in default. UseEbInvocation::args_with_configuredwhereEASYBUILD_ROBOT_PATHSor a configuration file may set them.
- fn args_with_configured(&self, configured: &[String], extra: &[String]) -> Result<Vec<String>, String>¶
EbInvocation::args, withrobotsearched beforeconfigured, the robot paths EasyBuild is configured with (configured_robot_paths).--robot-pathsreplaces the configured value, and an empty element in it stands for EasyBuild’s built-in default only, not for a value set inEASYBUILD_ROBOT_PATHSor a configuration file. So the configured paths are spelled out afterrobot. Withoutrobotthe arguments carry no--robot-pathsand EasyBuild keeps its own.
- fn dry_run_args(&self, dir: &Path) -> Vec<String>¶
eb -xarguments for this invocation, with the temporary files kept in <dir>/tmp, asEbInvocation::argsgives a real build. EasyBuild 5.4.1 can exit 1 after a dry run that succeeded, while it removes a temporary log that is already gone (ERROR: Failed to remove log file(s) … [Errno 2]` afterBuild succeeded for 1 out of 1`), and a dry run that exits 1 has its findings dropped. Builds already pass ``--disable-cleanup-tmpdir; dry runs pass it too. OpenMPI’s own removal of$TMPDIRis refused bydry_run_guard, not by this flag.
- fn dry_run_args_with_configured(&self, dir: &Path, configured: &[String]) -> Result<Vec<String>, String>¶
EbInvocation::dry_run_args, withrobotsearched beforeconfigured, the robot paths EasyBuild is configured with (configured_robot_paths).
- fn resolved(&self, root: &Path) -> Result<EbInvocation, String>¶
Check the invocation against the workspace: the recipe and every easyblock resolve inside
root, easyblocks are.pyfiles, and each--try-amendvalue isname=valuewith a lowercase parameter name.
- struct EbRun¶
How one
ebrun ended.- argv: Vec<String>¶
The command line, program first.
- exit_code: Option<i32>¶
Exit code;
Nonewhen killed.
- timed_out: bool¶
True when the run was killed at its time limit.
- seconds: u64¶
Wall time in seconds.
- output_file: PathBuf¶
Where stdout and stderr went.
- struct LogStep¶
One step’s span in a log:
firstandendare 0-based line indexes,endexclusive.- step: String¶
Step name (
configure,test, …).
- first_line: usize¶
First line of the step, 1-based.
- lines: usize¶
Number of lines in the step.
- struct ThreadedFinding¶
A generated command that carries the value of a plain library variable while the recipe turns a threaded toolchain option on, and lacks what the threaded variant of that variable adds.
- step: String¶
The step the command runs in.
- command: String¶
The command, at most 300 characters.
- plain: Vec<String>¶
Plain variables whose value the command carries.
- threaded: Vec<String>¶
The threaded variable of each entry of
plain, in the same order.
- missing: Vec<String>¶
Libraries the threaded variants add that the command does not have.
- struct ThreadedPair¶
A build-environment variable and its threaded variant, when their values differ.
- plain: String¶
The plain name (
LIBX,X_STATIC_LIBS).
- threaded: String¶
The threaded name (
LIBX_MT,X_MT_STATIC_LIBS,X_STATIC_LIBS_MT).
- plain_value: String¶
Value of the plain name.
- threaded_value: String¶
Value of the threaded name.