mod inode_budget

module inode_budget

Plan builds around a scratch inode quota.

A native EasyBuild job keeps its build tree on the scratch file system while it runs. A regression-test-heavy code unpacks several hundred thousand files; seven such jobs at once put a campaign over a quota of three million inodes, and an admission check that only looks at the free count at start time admits one job too many. This module gives the planner and the controller one footprint model so the cap is decided when the shards are planned, and the check at admission time counts what the running jobs have reserved.

Three pieces:

  • A footprint table, footprints.json: per software name, the peak build-tree inode count. derive takes the peak over ledger records whose build.inodes_peak names the count a build reached, a static table for packages the ledger has not seen, and the bundled classes (classes) for the rest. Footprints::of_module answers for a module or an easyconfig filename.

  • A shard footprint, shard_footprint: a shard builds its modules one after another, so one build tree is on disk at a time, and the shard costs its largest module footprint plus DEFAULT_LINE_OVERHEAD inodes per line for the tarballs, the EasyBuild tmpdir, and the logs the line leaves on scratch until the job ends. max_concurrent is the most shards whose footprints fit under the quota minus a margin at once, taking the largest first.

  • An admission decision, decide: the inodes the quota still allows, minus what the jobs that are still running have reserved, must cover this job’s footprint and the margin. The reservation is a file per job under one directory; a reservation whose job the caller does not list as live is stale: ignored, and removed only on request.

Variables

const BUNDLED_CLASSES: &str

The bundled footprint classes, data/inode-footprint-classes.toml: packages no record or table names, by software name.

const DEFAULT_FOOTPRINT: u64

Build-tree inodes assumed for a package no record, table, or class names.

const DEFAULT_LINE_OVERHEAD: u64

Inodes one build-list line keeps on scratch beside its build tree: the source tarballs, the EasyBuild tmpdir, and the logs, until the job ends.

const DEFAULT_MARGIN: u64

Inodes kept free under the quota by default.

const REFUSED_EXIT: i32

Exit status of stack admit when the job is refused.

Functions

fn check_job(job: &str) -> Result<()>

A job id is a file name: non-empty, no /, not . or .., and only letters, digits, ., _, and -.

fn classes() -> anyhow::Result<BTreeMap<String, u64>>

The bundled footprint classes as a map from software name to inodes.

Errors

The bundled table is not valid TOML with a [classes] table of integers.

fn decide(available: u64, reservations: &[Reservation], job: &str, live: Option<&[String]>, footprint: u64, margin: u64, unknown_reserve: u64) -> Decision

Decide admission. live is the jobs still running, as the caller’s scheduler lists them; absent, every reservation but this job’s own is counted and none is stale. A live job with no reservation is an unknown writer and counts unknown_reserve inodes. This job’s own earlier reservation is never counted, so a job that asks again is judged on the others alone.

fn derive(entries: &[crate::ledger::Entry], table: &BTreeMap<String, u64>) -> (Footprints, BTreeMap<String, Source>)

Derive the footprint table: the ledger peak and the static table where either names a package (the larger when both do), then the bundled classes (classes) for the rest, over DEFAULT_FOOTPRINT.

fn inodes_peak(record: &Map<String, Value>) -> Option<u64>

build.inodes_peak of a record, when build is an object carrying an integer of that name.

fn ledger_peaks(entries: &[crate::ledger::Entry]) -> BTreeMap<String, (u64, String, usize)>

Peak build.inodes_peak per software name over ledger records: the peak, the module that reached it, and the records that carried a value.

fn max_concurrent(footprints: &[u64], budget: u64) -> usize

The most shards that fit under budget at once: the largest k footprints sum to at most budget.

fn package_of_module(module: &str) -> &str

The software name of a module Name/VersionFull.

fn parse_quota(value: &Value) -> Result<Quota>

Read a quota document: inodes_available, or the limit that governs minus inodes_used. Which limit governs follows GPFS: below the soft limit (inodes_grace absent or none) the soft limit is the one to stay under; above it with the grace period running (inodes_grace such as 6 days) writes are allowed up to the hard limit, so inodes_hard governs; once the grace has expired the soft limit is enforced and nothing is available. inodes_hard and inodes_grace are carried through.

fn parse_table(text: &str) -> Result<BTreeMap<String, u64>, String>

Parse a static footprint table: Name<TAB>inodes per line (a space also separates), # comments and blank lines skipped. A later line for the same name replaces an earlier one.

fn read_footprints(path: &Path) -> Result<Footprints>

Read a footprints.json.

fn read_reservations(dir: &Path) -> Result<Vec<Reservation>>

Read every *.json reservation under dir. A missing directory is no reservations. A file that is not a reservation is an error, because a reservation that cannot be read hides inodes.

fn run_stack_admit(args: &AdmitRequest) -> Result<AdmitReport>

Under the lock when one is named: observe the quota, read the reservations, decide, write the reservation on admit, and remove stale reservations only when asked. A failed quota observation refuses.

fn run_stack_footprints(args: &FootprintsRequest) -> Result<FootprintsRun>

Read ledgers and tables and write footprints.json.

fn shard_footprint(modules: &[String], lines: usize, footprints: &Footprints, line_overhead: u64) -> ShardFootprint

The footprint of a shard that builds modules over lines build-list lines: the largest module footprint plus line_overhead per line. Modules build one after another, so one build tree is on disk at a time.

Enums

enum LockKind

Which kind of lock --lock takes. The two are separate namespaces on Linux: a controller that holds a POSIX record lock (fcntl, lockf) on the file is not excluded by a flock, and the other way round.

Flock

BSD flock: shared by processes and threads holding the file.

Posix

POSIX record lock over the whole file (fcntl F_SETLKW), the lock Python’s fcntl.lockf takes; held per process.

Implementations

impl LockKind

Functions

fn name(self) -> &'static str

The name on the command line and in the JSON line.

fn parse(name: &str) -> Result<Self>

Parse the command-line name.

Errors

A name that is neither flock nor posix.

enum Source

Where a package’s footprint came from.

Ledger

The peak build.inodes_peak over ledger records, with the module that reached it and the records counted.

module: String

The module whose record reached the peak.

records: usize

How many ledger records carried a peak for the package.

Table

A static table line.

LedgerAndTable

Both; the larger won.

module: String

The module whose record reached the ledger peak.

records: usize

How many ledger records carried a peak for the package.

Class

A the bundled classes (classes) entry.

Implementations

impl Source

Structs and Unions

struct AdmitReport

What stack admit decided and wrote.

decision: Decision

The decision.

reservation: Option<PathBuf>

The reservation written, when admitted.

line: String

The one JSON line to print.

struct AdmitRequest

Arguments for stack admit.

quota_json: Option<PathBuf>

The quota document, or - for stdin; absent with quota_cmd.

quota_cmd: Option<String>

A shell command whose standard output is the quota document, run under the lock so the observation and the decision are one step. A command that fails refuses the job instead of admitting it.

lock: Option<PathBuf>

A lock file held exclusively from the quota observation to the reservation write, so two controllers cannot both admit.

lock_kind: LockKind

Which lock the file takes; it must match what the other holders take.

unknown_reserve: u64

Inodes counted for each live job without a reservation.

drop_stale: bool

Remove stale reservation files; otherwise they are ignored and kept.

footprint: u64

This job’s footprint, from plan.json.

job: String

This job’s id.

reservations: PathBuf

The reservation directory.

margin: u64

Inodes kept free.

live_jobs: Option<Vec<String>>

Jobs still running, from the caller’s scheduler; absent counts every other reservation. A list expanded before the lock is taken can miss a job that reserved in between; live_jobs_cmd takes the snapshot inside the lock instead.

live_jobs_cmd: Option<String>

A shell command that prints the running job ids (one per line or separated by commas or spaces), run inside the lock after the quota observation. A command that fails refuses the job.

ts: Option<String>

RFC 3339 time written into the reservation; absent is now.

struct Decision

The admission decision.

admitted: bool

Whether the job may start.

available: u64

Inodes the quota still allows.

reserved: u64

Inodes the live jobs have reserved, this job’s own left out.

footprint: u64

This job’s footprint.

margin: u64

The margin kept free.

headroom: i64

available - reserved - footprint - margin; negative when refused.

counted: Vec<Reservation>

Reservations counted, by job.

stale: Vec<Reservation>

Reservations of jobs not in the live list: stale, not counted.

unknown: Vec<String>

Live jobs with no reservation, this job left out.

unknown_reserved: u64

Inodes counted for the unknown jobs.

struct Footprints

Peak build-tree inodes per software name.

default: u64

The footprint of a package packages does not name.

packages: BTreeMap<String, u64>

Footprint per software name.

Implementations

impl Footprints

Functions

fn builtin() -> Self

The bundled classes over DEFAULT_FOOTPRINT.

Panics

The bundled class table does not parse; a unit test keeps it valid.

fn from_json(value: &Value) -> Result<Self>

Read a footprints.json: the document stack footprints writes (footprints object and default), or a flat object of name: inodes, which a hand-kept file is.

fn of_module(&self, module: &str) -> u64

The footprint of a module Name/VersionFull, or of an easyconfig filename Name-version-toolchain.eb when the line was planned opaque. A filename matches the longest table name it starts with, followed by -, so jupyter-server-2.20.0-GCCcore-15.2.0.eb is jupyter-server.

fn of_package(&self, name: &str) -> u64

The footprint of a software name.

Traits implemented

impl Default for Footprints
struct FootprintsRequest

Arguments for stack footprints.

ledgers: Vec<PathBuf>

Ledger JSON Lines files.

tables: Vec<PathBuf>

Static tables, Name<TAB>inodes per line; a later file wins a name.

out: PathBuf

The footprints.json to write.

generated_at: Option<String>

RFC 3339 time written into the file; absent is now.

struct FootprintsRun

What stack footprints wrote.

packages: usize

Packages in the table.

from_ledger: usize

Packages whose footprint a ledger record set.

from_table: usize

Packages whose footprint a static table set.

from_class: usize

Packages taken from the bundled classes (classes).

records: usize

Ledger records that carried build.inodes_peak.

Implementations

impl FootprintsRun

Functions

fn summary(&self) -> String

One line for the CLI.

struct PlanBudget

The inode budget of a shard plan, for plan.json.

quota: Option<u64>

--inode-quota, when given.

margin: u64

--inode-margin.

footprints_file: Option<PathBuf>

The footprints file, when given.

footprints_sha256: Option<String>

Its SHA-256, when given.

footprints: Footprints

The footprint table in force.

line_overhead: u64

--inode-line-overhead.

Implementations

impl PlanBudget

Functions

fn apply(&self, shards: &[crate::shard::Shard], prelude: &[String], shard_name: impl Fn(usize) -> String) -> Result<(Vec<Value>, Value, usize)>

The footprint of each shard, the prelude’s, and the cap. The cap is the shard count when no quota is given. An error names a shard that does not fit under the budget on its own.

fn apply_parts(&self, shards: &[(Vec<String>, usize)], prelude: &[String], shard_name: impl Fn(usize) -> String) -> Result<(Vec<Value>, Value, usize)>

Self::apply over shards given as their modules and line counts, for a planner that has already moved lines between shards.

Errors

A shard that does not fit under the quota minus the margin on its own.

fn budget(&self) -> Option<u64>

quota - margin, when a quota is given.

fn to_json(&self) -> Value

The inode_budget object of plan.json.

struct Quota

What the quota file says.

available: u64

Inodes the quota still allows.

used: Option<u64>

inodes_used, when given.

soft: Option<u64>

inodes_soft, when given.

hard: Option<u64>

inodes_hard, when given.

grace: Option<String>

inodes_grace, when given: GPFS’s grace column, none below the soft limit, a remaining time while the grace period runs, expired after.

struct Reservation

One job’s reservation.

job: String

The job id.

footprint: u64

Inodes reserved.

ts: String

When it was written, RFC 3339.

struct ShardFootprint

The inodes one shard holds on scratch while it runs.

inodes: u64

peak + overhead.

peak: u64

The largest module footprint in the shard.

peak_module: Option<String>

The module with that footprint, when the shard builds anything.

overhead: u64

The line overhead summed over the shard’s lines.

Implementations

impl ShardFootprint