Repair a failed campaign¶
Use this procedure when eb-stack campaign run records a failed attempt. The
campaign state already contains the routed command, compact nested log, error
class, repair disposition, and ownership queue. Keep that evidence attached to
the same state file through every retry.
Inspect the active finding¶
Start with a compact view, then read the complete finding before changing a recipe or target:
The bundle is the one Author a new package writes at
/tmp/eb-stack/bundles/eon. The target below names an EasyBuild program
that is not on PATH, so the attempt fails at once and the finding id
is attempt:1:finding:1.
eb-stack campaign run \
--bundle /tmp/eb-stack/bundles/eon \
--config examples/targets/missing-easybuild.toml \
--target missing-eb \
--state /tmp/eb-stack/bundles/eon-repair.campaign.json
eb-stack campaign status --state /tmp/eb-stack/bundles/eon-repair.campaign.json |
jq '{status, attempts, claims, current_recipe,
active_findings: [.findings[] |
select(.status == "open" or .status == "in-progress") |
{id, class, disposition, stage, recipe, summary, owner,
command, exit_code, evidence}]}'
campaign run exits 1 and prints campaign build failed with typed findings.
The status document has status = failed, resolves = true, builds = false,
and one open finding of class runtime. The evidence names
/definitely/missing/eb-stack-easybuild.
The three claims are independent. A failed build can retain resolves: true
while builds and binary_verified remain false. Resolving a finding records
a repair decision; only a successful campaign retry establishes a build claim.
Check the routing layers¶
Run the target doctor for transport, executor, or runtime findings, and
before diagnosing a product failure on a target whose configuration changed:
eb-stack target doctor \
--config examples/targets/missing-easybuild.toml \
--target missing-eb
Doctor exits 1. stderr is:
error: target missing-eb doctor failed
Stdout is the check JSON. The easybuild check fails, and its stderr
field is:
"stderr": "spawn target command /definitely/missing/eb-stack-easybuild: No such file or directory (os error 2)"
It does not build the recipe. Compare that program with the failed finding. A healthy local target is:
eb-stack target doctor \
--config examples/targets/local-podman.toml \
--target local-rocky9
Container build and install roots belong to one runtime ABI. Keep Cargo and other package-manager build directories outside bind-mounted personal configuration ancestry. The target reference includes a complete neutral mount-alias pattern for sites whose durable storage lives below a home directory.
Claim the finding¶
Use a stable worker name. Claims prevent Hermes and OMP workers from applying different repairs to the same evidence:
eb-stack campaign finding claim \
--state /tmp/eb-stack/bundles/eon-repair.campaign.json \
--id attempt:1:finding:1 \
--owner hermes
The same owner may claim again safely. Another owner receives an ownership error and must select a different open finding.
Choose the repair surface¶
Treat the class as the first place to inspect, not as a substitute for the compiler, linker, test, or runtime evidence.
Finding class |
First check |
Typical changed artifact |
|---|---|---|
|
Host reachability and staged paths |
Private site target layer |
|
Scheduler command and allocation |
Private site target layer |
|
Runtime executable, image, mounts, ABI, environment |
Target TOML or container definition |
|
Absence of the prior routed workload and ownership of the campaign lock |
No artifact for a clean retry; target repair when a workload survived |
|
Upstream URL, mirror, cache, and target-runtime reachability |
Recipe source or target runtime |
|
Source/patch order and upstream digest |
Easyconfig checksum list |
|
Patch source, order, and applicability |
Patch plus Easyconfig |
|
Manifest/profile requirement and robot candidates |
Profile, stack policy, or dependency recipe |
|
Product options and detected dependencies |
Easyconfig, profile, or patch |
|
First source-level diagnostic and effective flags |
Easyconfig, patch, dependency, or target environment |
|
ABI, selected libraries, and link interface |
Easyconfig, patch, dependency, or target environment |
|
Failing product assertion and runtime environment |
Product repair or target environment |
|
Install paths, permissions, and EasyBlock behavior |
Easyconfig or target filesystem |
|
EULA refusal and the dependency that carries it |
Site decision on |
|
Declared files, commands, and loaded module |
Easyconfig or profile verification command |
|
Kernel/scheduler evidence and parallelism |
Executor allocation or |
|
Routed timeout and scheduler wall time |
Executor or target policy |
|
Exact command and nested log |
Evidence-dependent |
A solver-compatible selection is not a promise that source code compiles. A
real compiler incompatibility remains a compile repair owned by the campaign
loop. Add a stack exclusion or change a pin only when the evidence identifies
the selected dependency identity as the cause.
For recipe changes, run the nearest mechanical checks before the campaign:
eb-stack recipe format /tmp/eb-stack/bundles/eon/easyconfigs/e/eOn/*.eb
eb-stack recipe lint /tmp/eb-stack/bundles/eon/easyconfigs/e/eOn/*.eb
eb-stack recipe check \
--recipe /tmp/eb-stack/bundles/eon/easyconfigs/e/eOn/eOn-2.16.0-foss-2026.1.eb \
--easyconfigs /tmp/src/easybuild-easyconfigs/easybuild/easyconfigs \
--easyconfigs fixtures/eon_core_rgpot/easyconfigs \
--easyconfigs /tmp/eb-stack/bundles/eon/easyconfigs
Checksum lists are positional: every source first, then every patch. A missing dependency hint is a repair queue, including candidates at other generations; it is not permission to bypass recipe validation.
Build failures are not package parser policy. Classify with
BuildFindingClass, claim ownership, and record a FindingResolution
(action, evidence, changed paths). Do not invent package-specific parser
branches or arbitrary manifest mutation tools to clear a finding.
Record the repair¶
Resolve the owned finding with an action, evidence from the nearest check, and every changed artifact:
eb-stack campaign finding resolve \
--state /tmp/eb-stack/bundles/eon-repair.campaign.json \
--id attempt:1:finding:1 \
--owner hermes \
--action "point the EasyBuild command at the program the target image provides" \
--evidence "target doctor easybuild check names the missing program" \
--change examples/targets/local-podman.toml
The resolution is an audit record. The finding status becomes resolved
and the owner stays hermes.
Retry the same state¶
Run the original bundle and target with the same state path:
eb-stack campaign run \
--bundle /tmp/eb-stack/bundles/eon \
--config examples/targets/local-podman.toml \
--target local-rocky9 \
--state /tmp/eb-stack/bundles/eon-repair.campaign.json
The command stays in the foreground. Run it under the site’s terminal or
service supervisor and inspect status from another shell. A successful attempt
retains resolved findings and supersedes matching open findings, then advances
builds and binary_verified only when their commands succeed.
If the controller exits, its advisory state guard releases automatically. Confirm that its container, scheduler job, or remote command is absent before retrying because a routed workload can outlive the controller. Reuse the state file so attempt history and evidence remain one coherent record.