mod inode_budget¶
- module inode_budget¶
Plan builds around a scratch inode quota.
A native EasyBuild job keeps its build tree on the scratch file system while it runs. A regression-test-heavy code unpacks several hundred thousand files; seven such jobs at once put a campaign over a quota of three million inodes, and an admission check that only looks at the free count at start time admits one job too many. This module gives the planner and the controller one footprint model so the cap is decided when the shards are planned, and the check at admission time counts what the running jobs have reserved.
Three pieces:
A footprint table,
footprints.json: per software name, the peak build-tree inode count.derivetakes the peak over ledger records whosebuild.inodes_peaknames the count a build reached, a static table for packages the ledger has not seen, and the bundled classes (classes) for the rest.Footprints::of_moduleanswers for a module or an easyconfig filename.A shard footprint,
shard_footprint: a shard builds its modules one after another, so one build tree is on disk at a time, and the shard costs its largest module footprint plusDEFAULT_LINE_OVERHEADinodes per line for the tarballs, the EasyBuild tmpdir, and the logs the line leaves on scratch until the job ends.max_concurrentis the most shards whose footprints fit under the quota minus a margin at once, taking the largest first.An admission decision,
decide: the inodes the quota still allows, minus what the jobs that are still running have reserved, must cover this job’s footprint and the margin. The reservation is a file per job under one directory; a reservation whose job the caller does not list as live is stale: ignored, and removed only on request.
Variables
- const BUNDLED_CLASSES: &str¶
The bundled footprint classes,
data/inode-footprint-classes.toml: packages no record or table names, by software name.
- const DEFAULT_FOOTPRINT: u64¶
Build-tree inodes assumed for a package no record, table, or class names.
- const DEFAULT_LINE_OVERHEAD: u64¶
Inodes one build-list line keeps on scratch beside its build tree: the source tarballs, the EasyBuild tmpdir, and the logs, until the job ends.
- const DEFAULT_MARGIN: u64¶
Inodes kept free under the quota by default.
- const REFUSED_EXIT: i32¶
Exit status of
stack admitwhen the job is refused.
Functions
- fn check_job(job: &str) -> Result<()>¶
A job id is a file name: non-empty, no
/, not.or.., and only letters, digits,.,_, and-.
- fn classes() -> anyhow::Result<BTreeMap<String, u64>>¶
The bundled footprint classes as a map from software name to inodes.
Errors
The bundled table is not valid TOML with a
[classes]table of integers.
- fn decide(available: u64, reservations: &[Reservation], job: &str, live: Option<&[String]>, footprint: u64, margin: u64, unknown_reserve: u64) -> Decision¶
Decide admission.
liveis the jobs still running, as the caller’s scheduler lists them; absent, every reservation but this job’s own is counted and none is stale. A live job with no reservation is an unknown writer and countsunknown_reserveinodes. This job’s own earlier reservation is never counted, so a job that asks again is judged on the others alone.
- fn derive(entries: &[crate::ledger::Entry], table: &BTreeMap<String, u64>) -> (Footprints, BTreeMap<String, Source>)¶
Derive the footprint table: the ledger peak and the static table where either names a package (the larger when both do), then the bundled classes (
classes) for the rest, overDEFAULT_FOOTPRINT.
- fn inodes_peak(record: &Map<String, Value>) -> Option<u64>¶
build.inodes_peakof a record, whenbuildis an object carrying an integer of that name.
- fn ledger_peaks(entries: &[crate::ledger::Entry]) -> BTreeMap<String, (u64, String, usize)>¶
Peak
build.inodes_peakper software name over ledger records: the peak, the module that reached it, and the records that carried a value.
- fn max_concurrent(footprints: &[u64], budget: u64) -> usize¶
The most shards that fit under
budgetat once: the largestkfootprints sum to at mostbudget.
- fn package_of_module(module: &str) -> &str¶
The software name of a module
Name/VersionFull.
- fn parse_quota(value: &Value) -> Result<Quota>¶
Read a quota document:
inodes_available, or the limit that governs minusinodes_used. Which limit governs follows GPFS: below the soft limit (inodes_graceabsent ornone) the soft limit is the one to stay under; above it with the grace period running (inodes_gracesuch as6 days) writes are allowed up to the hard limit, soinodes_hardgoverns; once the grace hasexpiredthe soft limit is enforced and nothing is available.inodes_hardandinodes_graceare carried through.
- fn parse_table(text: &str) -> Result<BTreeMap<String, u64>, String>¶
Parse a static footprint table: Name<TAB>inodes per line (a space also separates),
#comments and blank lines skipped. A later line for the same name replaces an earlier one.
- fn read_footprints(path: &Path) -> Result<Footprints>¶
Read a
footprints.json.
- fn read_reservations(dir: &Path) -> Result<Vec<Reservation>>¶
Read every
*.jsonreservation underdir. A missing directory is no reservations. A file that is not a reservation is an error, because a reservation that cannot be read hides inodes.
- fn run_stack_admit(args: &AdmitRequest) -> Result<AdmitReport>¶
Under the lock when one is named: observe the quota, read the reservations, decide, write the reservation on admit, and remove stale reservations only when asked. A failed quota observation refuses.
- fn run_stack_footprints(args: &FootprintsRequest) -> Result<FootprintsRun>¶
Read ledgers and tables and write
footprints.json.
- fn shard_footprint(modules: &[String], lines: usize, footprints: &Footprints, line_overhead: u64) -> ShardFootprint¶
The footprint of a shard that builds
modulesoverlinesbuild-list lines: the largest module footprint plusline_overheadper line. Modules build one after another, so one build tree is on disk at a time.
Enums
- enum LockKind¶
Which kind of lock
--locktakes. The two are separate namespaces on Linux: a controller that holds a POSIX record lock (fcntl,lockf) on the file is not excluded by aflock, and the other way round.- Flock¶
BSD
flock: shared by processes and threads holding the file.
- Posix¶
POSIX record lock over the whole file (
fcntlF_SETLKW), the lock Python’sfcntl.lockftakes; held per process.
Implementations
- enum Source¶
Where a package’s footprint came from.
- Ledger¶
The peak
build.inodes_peakover ledger records, with the module that reached it and the records counted.- module: String¶
The module whose record reached the peak.
- records: usize¶
How many ledger records carried a peak for the package.
- Table¶
A static table line.
- LedgerAndTable¶
Both; the larger won.
- module: String¶
The module whose record reached the ledger peak.
- records: usize¶
How many ledger records carried a peak for the package.
- Class¶
A the bundled classes (
classes) entry.
Implementations
Structs and Unions
- struct AdmitReport¶
What
stack admitdecided and wrote.- reservation: Option<PathBuf>¶
The reservation written, when admitted.
- line: String¶
The one JSON line to print.
- struct AdmitRequest¶
Arguments for
stack admit.- quota_json: Option<PathBuf>¶
The quota document, or
-for stdin; absent withquota_cmd.
- quota_cmd: Option<String>¶
A shell command whose standard output is the quota document, run under the lock so the observation and the decision are one step. A command that fails refuses the job instead of admitting it.
- lock: Option<PathBuf>¶
A lock file held exclusively from the quota observation to the reservation write, so two controllers cannot both admit.
- unknown_reserve: u64¶
Inodes counted for each live job without a reservation.
- drop_stale: bool¶
Remove stale reservation files; otherwise they are ignored and kept.
- footprint: u64¶
This job’s footprint, from
plan.json.
- job: String¶
This job’s id.
- reservations: PathBuf¶
The reservation directory.
- margin: u64¶
Inodes kept free.
- live_jobs: Option<Vec<String>>¶
Jobs still running, from the caller’s scheduler; absent counts every other reservation. A list expanded before the lock is taken can miss a job that reserved in between;
live_jobs_cmdtakes the snapshot inside the lock instead.
- live_jobs_cmd: Option<String>¶
A shell command that prints the running job ids (one per line or separated by commas or spaces), run inside the lock after the quota observation. A command that fails refuses the job.
- ts: Option<String>¶
RFC 3339 time written into the reservation; absent is now.
- struct Decision¶
The admission decision.
- admitted: bool¶
Whether the job may start.
- available: u64¶
Inodes the quota still allows.
- reserved: u64¶
Inodes the live jobs have reserved, this job’s own left out.
- footprint: u64¶
This job’s footprint.
- margin: u64¶
The margin kept free.
- headroom: i64¶
available - reserved - footprint - margin; negative when refused.
- counted: Vec<Reservation>¶
Reservations counted, by job.
- stale: Vec<Reservation>¶
Reservations of jobs not in the live list: stale, not counted.
- unknown: Vec<String>¶
Live jobs with no reservation, this job left out.
- unknown_reserved: u64¶
Inodes counted for the unknown jobs.
- struct Footprints¶
Peak build-tree inodes per software name.
- default: u64¶
The footprint of a package
packagesdoes not name.
- packages: BTreeMap<String, u64>¶
Footprint per software name.
Implementations
- impl Footprints¶
Functions
- fn builtin() -> Self¶
The bundled classes over
DEFAULT_FOOTPRINT.Panics
The bundled class table does not parse; a unit test keeps it valid.
- fn from_json(value: &Value) -> Result<Self>¶
Read a
footprints.json: the documentstack footprintswrites (footprintsobject anddefault), or a flat object ofname: inodes, which a hand-kept file is.
- fn of_module(&self, module: &str) -> u64¶
The footprint of a module
Name/VersionFull, or of an easyconfig filenameName-version-toolchain.ebwhen the line was planned opaque. A filename matches the longest table name it starts with, followed by-, sojupyter-server-2.20.0-GCCcore-15.2.0.ebisjupyter-server.
- fn of_package(&self, name: &str) -> u64¶
The footprint of a software name.
Traits implemented
- impl Default for Footprints¶
- struct FootprintsRequest¶
Arguments for
stack footprints.- ledgers: Vec<PathBuf>¶
Ledger JSON Lines files.
- tables: Vec<PathBuf>¶
Static tables, Name<TAB>inodes per line; a later file wins a name.
- out: PathBuf¶
The
footprints.jsonto write.
- generated_at: Option<String>¶
RFC 3339 time written into the file; absent is now.
- struct FootprintsRun¶
What
stack footprintswrote.- packages: usize¶
Packages in the table.
- from_ledger: usize¶
Packages whose footprint a ledger record set.
- from_table: usize¶
Packages whose footprint a static table set.
- from_class: usize¶
Packages taken from the bundled classes (
classes).
- records: usize¶
Ledger records that carried
build.inodes_peak.
Implementations
- impl FootprintsRun¶
Functions
- fn summary(&self) -> String¶
One line for the CLI.
- struct PlanBudget¶
The inode budget of a shard plan, for
plan.json.- quota: Option<u64>¶
--inode-quota, when given.
- margin: u64¶
--inode-margin.
- footprints_file: Option<PathBuf>¶
The footprints file, when given.
- footprints_sha256: Option<String>¶
Its SHA-256, when given.
- footprints: Footprints¶
The footprint table in force.
- line_overhead: u64¶
--inode-line-overhead.
Implementations
- impl PlanBudget¶
Functions
- fn apply(&self, shards: &[crate::shard::Shard], prelude: &[String], shard_name: impl Fn(usize) -> String) -> Result<(Vec<Value>, Value, usize)>¶
The footprint of each shard, the prelude’s, and the cap. The cap is the shard count when no quota is given. An error names a shard that does not fit under the budget on its own.
- fn apply_parts(&self, shards: &[(Vec<String>, usize)], prelude: &[String], shard_name: impl Fn(usize) -> String) -> Result<(Vec<Value>, Value, usize)>¶
Self::applyover shards given as their modules and line counts, for a planner that has already moved lines between shards.Errors
A shard that does not fit under the quota minus the margin on its own.
- fn budget(&self) -> Option<u64>¶
quota - margin, when a quota is given.
- fn to_json(&self) -> Value¶
The
inode_budgetobject ofplan.json.
- struct Quota¶
What the quota file says.
- available: u64¶
Inodes the quota still allows.
- used: Option<u64>¶
inodes_used, when given.
- soft: Option<u64>¶
inodes_soft, when given.
- hard: Option<u64>¶
inodes_hard, when given.
- grace: Option<String>¶
inodes_grace, when given: GPFS’s grace column,nonebelow the soft limit, a remaining time while the grace period runs,expiredafter.
- struct Reservation¶
One job’s reservation.
- job: String¶
The job id.
- footprint: u64¶
Inodes reserved.
- ts: String¶
When it was written, RFC 3339.
- struct ShardFootprint¶
The inodes one shard holds on scratch while it runs.
- inodes: u64¶
peak + overhead.
- peak: u64¶
The largest module footprint in the shard.
- peak_module: Option<String>¶
The module with that footprint, when the shard builds anything.
- overhead: u64¶
The line overhead summed over the shard’s lines.
Implementations
- impl ShardFootprint¶