Repair a failed campaign

Use this procedure when eb-stack campaign run records a failed attempt. The campaign state already contains the routed command, compact nested log, error class, repair disposition, and ownership queue. Keep that evidence attached to the same state file through every retry.

Inspect the active finding

Start with a compact view, then read the complete finding before changing a recipe or target:

The bundle is the one Author a new package writes at /tmp/eb-stack/bundles/eon. The target below names an EasyBuild program that is not on PATH, so the attempt fails at once and the finding id is attempt:1:finding:1.

eb-stack campaign run \
  --bundle /tmp/eb-stack/bundles/eon \
  --config examples/targets/missing-easybuild.toml \
  --target missing-eb \
  --state /tmp/eb-stack/bundles/eon-repair.campaign.json

eb-stack campaign status --state /tmp/eb-stack/bundles/eon-repair.campaign.json |
  jq '{status, attempts, claims, current_recipe,
       active_findings: [.findings[] |
         select(.status == "open" or .status == "in-progress") |
         {id, class, disposition, stage, recipe, summary, owner,
          command, exit_code, evidence}]}'

campaign run exits 1 and prints campaign build failed with typed findings. The status document has status = failed, resolves = true, builds = false, and one open finding of class runtime. The evidence names /definitely/missing/eb-stack-easybuild.

The three claims are independent. A failed build can retain resolves: true while builds and binary_verified remain false. Resolving a finding records a repair decision; only a successful campaign retry establishes a build claim.

Check the routing layers

Run the target doctor for transport, executor, or runtime findings, and before diagnosing a product failure on a target whose configuration changed:

eb-stack target doctor \
  --config examples/targets/missing-easybuild.toml \
  --target missing-eb

Doctor exits 1. stderr is:

error: target missing-eb doctor failed

Stdout is the check JSON. The easybuild check fails, and its stderr field is:

"stderr": "spawn target command /definitely/missing/eb-stack-easybuild: No such file or directory (os error 2)"

It does not build the recipe. Compare that program with the failed finding. A healthy local target is:

eb-stack target doctor \
  --config examples/targets/local-podman.toml \
  --target local-rocky9

Container build and install roots belong to one runtime ABI. Keep Cargo and other package-manager build directories outside bind-mounted personal configuration ancestry. The target reference includes a complete neutral mount-alias pattern for sites whose durable storage lives below a home directory.

Claim the finding

Use a stable worker name. Claims prevent Hermes and OMP workers from applying different repairs to the same evidence:

eb-stack campaign finding claim \
  --state /tmp/eb-stack/bundles/eon-repair.campaign.json \
  --id attempt:1:finding:1 \
  --owner hermes

The same owner may claim again safely. Another owner receives an ownership error and must select a different open finding.

Choose the repair surface

Treat the class as the first place to inspect, not as a substitute for the compiler, linker, test, or runtime evidence.

Finding class

First check

Typical changed artifact

transport

Host reachability and staged paths

Private site target layer

executor

Scheduler command and allocation

Private site target layer

runtime

Runtime executable, image, mounts, ABI, environment

Target TOML or container definition

interrupted

Absence of the prior routed workload and ownership of the campaign lock

No artifact for a clean retry; target repair when a workload survived

source

Upstream URL, mirror, cache, and target-runtime reachability

Recipe source or target runtime

checksum

Source/patch order and upstream digest

Easyconfig checksum list

patch

Patch source, order, and applicability

Patch plus Easyconfig

dependency-missing

Manifest/profile requirement and robot candidates

Profile, stack policy, or dependency recipe

configure

Product options and detected dependencies

Easyconfig, profile, or patch

compile

First source-level diagnostic and effective flags

Easyconfig, patch, dependency, or target environment

link

ABI, selected libraries, and link interface

Easyconfig, patch, dependency, or target environment

test

Failing product assertion and runtime environment

Product repair or target environment

install

Install paths, permissions, and EasyBlock behavior

Easyconfig or target filesystem

license

EULA refusal and the dependency that carries it

Site decision on --accept-eula-for; nothing in the recipe

sanity

Declared files, commands, and loaded module

Easyconfig or profile verification command

resource

Kernel/scheduler evidence and parallelism

Executor allocation or EASYBUILD_PARALLEL

timeout

Routed timeout and scheduler wall time

Executor or target policy

unknown

Exact command and nested log

Evidence-dependent

A solver-compatible selection is not a promise that source code compiles. A real compiler incompatibility remains a compile repair owned by the campaign loop. Add a stack exclusion or change a pin only when the evidence identifies the selected dependency identity as the cause.

For recipe changes, run the nearest mechanical checks before the campaign:

eb-stack recipe format /tmp/eb-stack/bundles/eon/easyconfigs/e/eOn/*.eb
eb-stack recipe lint /tmp/eb-stack/bundles/eon/easyconfigs/e/eOn/*.eb
eb-stack recipe check \
  --recipe /tmp/eb-stack/bundles/eon/easyconfigs/e/eOn/eOn-2.16.0-foss-2026.1.eb \
  --easyconfigs /tmp/src/easybuild-easyconfigs/easybuild/easyconfigs \
  --easyconfigs fixtures/eon_core_rgpot/easyconfigs \
  --easyconfigs /tmp/eb-stack/bundles/eon/easyconfigs

Checksum lists are positional: every source first, then every patch. A missing dependency hint is a repair queue, including candidates at other generations; it is not permission to bypass recipe validation.

Build failures are not package parser policy. Classify with BuildFindingClass, claim ownership, and record a FindingResolution (action, evidence, changed paths). Do not invent package-specific parser branches or arbitrary manifest mutation tools to clear a finding.

Record the repair

Resolve the owned finding with an action, evidence from the nearest check, and every changed artifact:

eb-stack campaign finding resolve \
  --state /tmp/eb-stack/bundles/eon-repair.campaign.json \
  --id attempt:1:finding:1 \
  --owner hermes \
  --action "point the EasyBuild command at the program the target image provides" \
  --evidence "target doctor easybuild check names the missing program" \
  --change examples/targets/local-podman.toml

The resolution is an audit record. The finding status becomes resolved and the owner stays hermes.

Retry the same state

Run the original bundle and target with the same state path:

eb-stack campaign run \
  --bundle /tmp/eb-stack/bundles/eon \
  --config examples/targets/local-podman.toml \
  --target local-rocky9 \
  --state /tmp/eb-stack/bundles/eon-repair.campaign.json

The command stays in the foreground. Run it under the site’s terminal or service supervisor and inspect status from another shell. A successful attempt retains resolved findings and supersedes matching open findings, then advances builds and binary_verified only when their commands succeed.

If the controller exits, its advisory state guard releases automatically. Confirm that its container, scheduler job, or remote command is absent before retrying because a routed workload can outlive the controller. Reuse the state file so attempt history and evidence remain one coherent record.