mod shard

module shard

Split one Jenkins build list into shards that can build at the same time.

Two build-list lines interfere when the parts of their dependency closures that are not installed yet meet: both would build the same module. Each line’s closure is computed the way pr-overlap computes a root’s set, over the robot path minus the installed modules.

Lines are grouped first on root-level conflicts only: a line whose own easyconfig is in another line’s closure, or two lines naming the same easyconfig, have to build in one shard, because the line carries its own eb options and has to build that module itself. Any other module that the closures of two or more groups share moves to a prelude that builds before every shard. That keeps the groups as many and as small as the dependency graph allows. The prelude needs no line of its own: a module in it is never a line’s root (that line’s group would hold every closure that reaches it), and its dependencies are in every closure it is in, so they are in the prelude too.

Groups are then packed into at most shards shards by longest processing time first: heaviest group first, each onto the lightest shard, ties broken by fewer groups and then by shard number. A module weighs what --weights says it took to build, or the default weight.

Functions

fn is_commit_sha(value: &str) -> bool

A commit as --from-commit takes it: 7 to 40 hexadecimal digits.

fn merge_orders(orders: &[Vec<String>]) -> Vec<String>

Merge dependency orders into one order every input agrees with: a module comes after each module that precedes it in any input. A module goes next when it heads the first input that still has modules and no input has it further back. Orders that disagree, which closures over one robot path do not, fall back to the first input’s head.

fn parse_lines(text: &str) -> Vec<ShardLine>

Parse a build list. Blank lines and # comments are skipped; every other line is kept, whatever its first word, so a line that names no easyconfig is reported rather than dropped.

fn parse_weights(text: &str) -> Result<BTreeMap<String, u64>, String>

Read a weights file: module, a tab, then seconds per line. # comments and blank lines are skipped. A fractional value rounds up to whole seconds. A later line for the same module replaces an earlier one.

fn plan(closures: &[LineClosure], order: &[String], weight: &dyn Fn(&str) -> u64, shards: usize) -> ShardPlan

Plan the shards with only shared dependencies in the prelude.

order is a dependency order over every module the closures name; the prelude is written in it. weight gives a module’s weight.

fn plan_with(closures: &[LineClosure], order: &[String], weight: &dyn Fn(&str) -> u64, shards: usize, cpu_prerequisites: &BTreeSet<String>) -> ShardPlan

Plan the shards with an optional CPU-prerequisite lift set.

cpu_prerequisites is the opt-in set of CPU-only modules that must build under the CPU prefix before any accelerator work. They are removed from shard closures and from the shared prelude (so a shared CUDA package can stay in prelude under accelerator settings while HDF5 leaves for the CPU list). An empty set preserves legacy behaviour: the prelude is exactly the modules two or more groups share. A module that is any line’s root is not taken from this set into the CPU dependency list; callers place CPU-only root lines themselves.

Enums

enum LineSource

The easyconfig tree a fetched line is resolved against.

Commit(String)

--from-commit=SHA: the easyconfigs of that commit.

Prs(Vec<u64>)

--from-pr=N[,M...]: the easyconfig files those pull requests change, at their heads.

Invalid(String)

The option is there but its value cannot be used; the text says why.

Implementations

impl LineSource

Functions

fn option(&self) -> String

The option as EasyBuild takes it, --from-commit=SHA or --from-pr=N,M.

Structs and Unions

struct LineClosure

What one line would build.

root: String

The line’s own module, Name/VersionFull.

to_build: Vec<String>

Modules not installed yet, dependencies first, the root included unless it is installed.

struct Shard

One packed shard.

lines: Vec<usize>

Indices into the line list, in dependency order.

modules: Vec<String>

Modules the shard builds, sorted.

weight: u64

Sum of the module weights.

struct ShardLine

One non-comment line of a build list.

number: usize

1-based line number in the input.

text: String

The line as written, trailing whitespace dropped.

file: String

The first word, the easyconfig filename.

forced: bool

Whether the line forces a rebuild (--rebuild, --force or its short form -f), so its own module counts as not installed.

fetched: bool

Whether EasyBuild fetches the line’s easyconfig itself (--from-pr or --from-commit), so it need not be on the robot path.

source: Option<LineSource>

Where EasyBuild fetches the easyconfigs from, when fetched.

source_options: Vec<String>

The options that say where the line’s easyconfigs and easyblocks come from (--from-commit, --from-pr, --include-easyblocks-from-commit, --include-easyblocks-from-pr), as written. A prelude line for a module this line’s tree provides carries them, so it builds the same easyconfig with the same easyblock.

Implementations

impl ShardLine

Functions

fn filter_deps_option(&self) -> Option<&str>

A dependency-filter override that the list-wide closure cannot model. EasyBuild accepts unambiguous prefixes of its long option names.

struct ShardPlan

The shard plan.

group_of: Vec<usize>

Group of each line, numbered by the group’s first line.

shard_of: Vec<usize>

Shard of each line, 0-based.

prelude: Vec<String>

Modules two or more groups need, in order. With plan_with CPU lifts, only the modules that stay under accelerator settings (shared accelerator packages) remain here.

prelude_weight: u64

Sum of the prelude module weights.

cpu_prerequisites: Vec<String>

CPU-only prerequisites that must build under the CPU prefix before the accelerator prelude. Empty when the opt-in lift set is empty (legacy).

cpu_prerequisites_weight: u64

Sum of the CPU-prerequisite module weights.

shards: Vec<Shard>

The shards, heaviest first.