mod shard¶
- module shard¶
Split one Jenkins build list into shards that can build at the same time.
Two build-list lines interfere when the parts of their dependency closures that are not installed yet meet: both would build the same module. Each line’s closure is computed the way
pr-overlapcomputes a root’s set, over the robot path minus the installed modules.Lines are grouped first on root-level conflicts only: a line whose own easyconfig is in another line’s closure, or two lines naming the same easyconfig, have to build in one shard, because the line carries its own
eboptions and has to build that module itself. Any other module that the closures of two or more groups share moves to a prelude that builds before every shard. That keeps the groups as many and as small as the dependency graph allows. The prelude needs no line of its own: a module in it is never a line’s root (that line’s group would hold every closure that reaches it), and its dependencies are in every closure it is in, so they are in the prelude too.Groups are then packed into at most
shardsshards by longest processing time first: heaviest group first, each onto the lightest shard, ties broken by fewer groups and then by shard number. A module weighs what--weightssays it took to build, or the default weight.Functions
- fn is_commit_sha(value: &str) -> bool¶
A commit as
--from-committakes it: 7 to 40 hexadecimal digits.
- fn merge_orders(orders: &[Vec<String>]) -> Vec<String>¶
Merge dependency orders into one order every input agrees with: a module comes after each module that precedes it in any input. A module goes next when it heads the first input that still has modules and no input has it further back. Orders that disagree, which closures over one robot path do not, fall back to the first input’s head.
- fn parse_lines(text: &str) -> Vec<ShardLine>¶
Parse a build list. Blank lines and
#comments are skipped; every other line is kept, whatever its first word, so a line that names no easyconfig is reported rather than dropped.
- fn parse_weights(text: &str) -> Result<BTreeMap<String, u64>, String>¶
Read a weights file:
module, a tab, thensecondsper line.#comments and blank lines are skipped. A fractional value rounds up to whole seconds. A later line for the same module replaces an earlier one.
- fn plan(closures: &[LineClosure], order: &[String], weight: &dyn Fn(&str) -> u64, shards: usize) -> ShardPlan¶
Plan the shards with only shared dependencies in the prelude.
orderis a dependency order over every module the closures name; the prelude is written in it.weightgives a module’s weight.
- fn plan_with(closures: &[LineClosure], order: &[String], weight: &dyn Fn(&str) -> u64, shards: usize, cpu_prerequisites: &BTreeSet<String>) -> ShardPlan¶
Plan the shards with an optional CPU-prerequisite lift set.
cpu_prerequisitesis the opt-in set of CPU-only modules that must build under the CPU prefix before any accelerator work. They are removed from shard closures and from the shared prelude (so a shared CUDA package can stay inpreludeunder accelerator settings while HDF5 leaves for the CPU list). An empty set preserves legacy behaviour: the prelude is exactly the modules two or more groups share. A module that is any line’s root is not taken from this set into the CPU dependency list; callers place CPU-only root lines themselves.
Enums
- enum LineSource¶
The easyconfig tree a fetched line is resolved against.
- Commit(String)¶
--from-commit=SHA: the easyconfigs of that commit.
- Prs(Vec<u64>)¶
--from-pr=N[,M...]: the easyconfig files those pull requests change, at their heads.
- Invalid(String)¶
The option is there but its value cannot be used; the text says why.
Implementations
- impl LineSource¶
Functions
- fn option(&self) -> String¶
The option as EasyBuild takes it,
--from-commit=SHAor--from-pr=N,M.
Structs and Unions
- struct LineClosure¶
What one line would build.
- root: String¶
The line’s own module,
Name/VersionFull.
- to_build: Vec<String>¶
Modules not installed yet, dependencies first, the root included unless it is installed.
- struct Shard¶
One packed shard.
- lines: Vec<usize>¶
Indices into the line list, in dependency order.
- modules: Vec<String>¶
Modules the shard builds, sorted.
- weight: u64¶
Sum of the module weights.
- struct ShardLine¶
One non-comment line of a build list.
- number: usize¶
1-based line number in the input.
- text: String¶
The line as written, trailing whitespace dropped.
- file: String¶
The first word, the easyconfig filename.
- forced: bool¶
Whether the line forces a rebuild (
--rebuild,--forceor its short form-f), so its own module counts as not installed.
- fetched: bool¶
Whether EasyBuild fetches the line’s easyconfig itself (
--from-pror--from-commit), so it need not be on the robot path.
- source: Option<LineSource>¶
Where EasyBuild fetches the easyconfigs from, when
fetched.
- source_options: Vec<String>¶
The options that say where the line’s easyconfigs and easyblocks come from (
--from-commit,--from-pr,--include-easyblocks-from-commit,--include-easyblocks-from-pr), as written. A prelude line for a module this line’s tree provides carries them, so it builds the same easyconfig with the same easyblock.
Implementations
- struct ShardPlan¶
The shard plan.
- group_of: Vec<usize>¶
Group of each line, numbered by the group’s first line.
- shard_of: Vec<usize>¶
Shard of each line, 0-based.
- prelude: Vec<String>¶
Modules two or more groups need, in
order. Withplan_withCPU lifts, only the modules that stay under accelerator settings (shared accelerator packages) remain here.
- prelude_weight: u64¶
Sum of the prelude module weights.
- cpu_prerequisites: Vec<String>¶
CPU-only prerequisites that must build under the CPU prefix before the accelerator prelude. Empty when the opt-in lift set is empty (legacy).
- cpu_prerequisites_weight: u64¶
Sum of the CPU-prerequisite module weights.