Campaign state and typed findings #

A campaign turns a resolved package bundle into empirical build and binary evidence on one named target. The JSON state is durable, resumable, and the coordination surface for Hermes and OMP workers.

Run and inspect #

eb-stack campaign run \
  --bundle BUNDLE \
  --config BASE.toml \
  --config SITE.toml \
  --target NAME \
  --state CAMPAIGN.json

eb-stack campaign status --state CAMPAIGN.json

campaign run is synchronous and stays in the foreground until every recipe and verification command finishes or a typed failure is recorded. Run it under the target site’s terminal or service supervisor; campaign status remains available from another process.

The state file has one writer while a run, claim, or resolution mutates it. An OS-backed advisory guard enforces exclusivity, and the visible .lock file records host, PID, and Linux process-start identity for diagnosis. The guard is released by the operating system if the controlling process exits, so stale metadata cannot permanently block a retry. Before retrying an interrupted run, confirm that the routed container, scheduler job, or host command is no longer active; a remote workload can outlive its controller. Reuse the same state path. Package name and version must match, each run increments attempts, history and findings remain available, and build/binary claims reset until the current attempt proves them.

campaign status prints the complete audit record. Typed failures retain their exact routed command, exit status, and compact evidence, including after their findings are resolved or superseded. For a compact operator view during a long build:

eb-stack campaign status --state CAMPAIGN.json |
  jq '{status, attempts, claims, current_recipe,
       active_findings: [.findings[] |
         select(.status == "open" or .status == "in-progress") |
         {id, class, stage, recipe, summary, owner}]}'

The campaign writes status: running and current_recipe before launching EasyBuild, so another shell can observe those fields while the build command is active. Keep the unfiltered state for diagnosis and handoff evidence.

State schema #

CAMPAIGN.json has schema_version = 1:

{
  "schema_version": 1,
  "package": "QMCPACK",
  "version": "4.3.0",
  "bundle": "/work/qmcpack",
  "target": "site-builder",
  "status": "failed",
  "attempts": 2,
  "claims": {
    "resolves": true,
    "builds": false,
    "binary_verified": false
  },
  "current_recipe": null,
  "findings": [],
  "history": []
}

Status is planned, running, failed, or completed. current_recipe is present while an EasyBuild recipe is active. History entries record attempt, status, optional recipe/profile, and a compact detail string.

Claim ladder #

The three booleans are independent claims:

resolves: every requested profile has a Resolvo lock and emitted recipe;
builds: every emitted recipe succeeds through EasyBuild on this target;
binary_verified: at least one verification command is declared and every declared command succeeds.

A completed campaign with no verification commands has builds: true and binary_verified: false. A planned SBOM never implies either build claim.

Finding schema #

A terminal failure appends a finding containing:

Field	Meaning
`id`	Stable `attempt:N:finding:M` identity
`class`	Error-domain classification
`disposition`	Mechanical, retryable, judgment, or target repair
`stage`	`preflight`, `stage`, `build`, or `verify`
`recipe`	Relative recipe path or verification profile
`target`	Routed target name
`summary` / `evidence`	Compact diagnosis and terminal evidence
`command` / `exit_code`	Exact routed program, arguments, and exit status
`attempt`	Attempt that observed the failure
`status` / `owner`	Queue state and exclusive worker owner
`resolution`	Action, evidence, and changed paths

Evidence contains the tail of routed stdout/stderr. When EasyBuild reports a separate combined-output path, the campaign interns the nested log before classification so the finding reflects the actual compiler or runtime error.

Classes, dispositions, and repair surfaces #

Class	Default disposition	Start at
`transport`	target-repair	SSH, rsync, host reachability, or staging path
`executor`	target-repair	Slurm command, partition, account, or allocation syntax
`runtime`	target-repair	Host/container executable, image, mounts, ABI, or environment
`checksum`	mechanical	Source digest and positional source/patch checksum list
`patch`	requires-judgment	Patch source, order, and applicability
`dependency-missing`	requires-judgment	Manifest, profile, stack policy, or real dependency recipe
`configure`	requires-judgment	Product options, detected dependency, toolchain, or patch
`compile`	requires-judgment	Source compatibility, compiler, flags, dependency, or patch
`link`	requires-judgment	Link interface, library order, ABI, or dependency selection
`test`	requires-judgment	Product behavior, test environment, or test-specific dependency
`install`	requires-judgment	EasyBuild install logic, paths, or permissions
`sanity`	requires-judgment	Sanity paths or profile verification command
`resource`	retryable	CPU, memory, GPU, filesystem, or scheduler sizing
`timeout`	retryable	Routed timeout or scheduler wall time
`unknown`	requires-judgment	Unclassified evidence; inspect the exact command and nested log

The class narrows the first repair surface; it does not replace diagnosis. For example, a SAT-compatible dependency that fails to compile remains a compile finding until evidence supports a stack exclusion or package patch. A compiler killed by the kernel, virtual memory exhausted, or equivalent allocation evidence is instead a resource finding. Preserve the selected dependency and recipe while correcting EASYBUILD_PARALLEL or the scheduler allocation; only source-level compiler diagnostics justify a compile repair.

Ownership protocol #

Claim an open finding before changing state shared with another worker:

eb-stack campaign finding claim \
  --state CAMPAIGN.json \
  --id attempt:2:finding:1 \
  --owner omp-worker-1

Claims are exclusive. Reclaiming with the same owner is idempotent; a different owner receives an ownership error.

Record the repair and its evidence:

eb-stack campaign finding resolve \
  --state CAMPAIGN.json \
  --id attempt:2:finding:1 \
  --owner omp-worker-1 \
  --action "corrected the container runtime environment" \
  --evidence "target doctor and the isolated command exit successfully" \
  --change ~/.config/eb-stack/site.toml

Run the campaign again with the same state. A successful retry marks matching open or in-progress findings as superseded while retaining their evidence. Resolved findings remain part of the audit trail.

Repair loop #

Run target doctor and the campaign.
Read the finding’s stage, class, exact command, evidence, and nested log.
Reproduce the smallest failing command on the same target/runtime.
Claim the finding.
Change the layer supported by the evidence.
Run the nearest mechanical check: recipe lint/check, target doctor, or the isolated build command.
Resolve the finding with concrete evidence and changed paths.
Rerun the full campaign state until every requested recipe and verification command succeeds.

Hermes owns this end-to-end loop. OMP workers use finding ownership to divide residual diagnosis without racing on the state file. The executable procedures are new package and annual bump.