feat(harbor): pick the runtime image per run; tag builds by commit

[ShuxinLin]

Description

You can now choose, for each run, which runtime image the Harbor tasks build FROM, and every runtime build gets a tag that pins it.

Before this PR, the base was fixed by the task Dockerfile's ARG default, assetopsbench/runtime:dev, and every rebuild overwrote that tag. Using a published image meant retagging it locally, and a finished run couldn't name the build it ran on.

1. Choosing the image per run (e91aa07)

• template/environment/docker-compose.yaml passes AOB_RUNTIME_IMAGE to the task Dockerfile's ARG as a build arg.
  • Harbor runs docker compose build with the shell's environment and every compose file, so AOB_RUNTIME_IMAGE=<image> harbor run … builds FROM that image.
  • The tasks don't need regenerating.
  • When the variable is unset, the local assetopsbench/runtime:dev is used, as before.
• run.sh -r IMAGE (the default comes from AOB_RUNTIME_IMAGE):
  • A registry reference such as quay.io/… or localhost:5000/… is pulled first. Otherwise a stale local copy would satisfy FROM.
  • A bare name must already exist locally, as before.
  • It exports the variable, so harbor jobs resume uses it too, and it prints the image id it runs on.

2. Tagging builds by commit (3bb4f12)

• build-runtime-image.sh: when no -t is given, it tags both assetopsbench/runtime:dev and assetopsbench/runtime:<short commit>.
  • :dev moves with every build. The commit tag never does, so -r assetopsbench/runtime:<commit> pins a run to one build.
  • The tag is always the right commit, because the image is built from git archive HEAD whatever the working tree holds.
• publish-images.sh pushes <namespace>/runtime:<commit> next to :TAG and :latest.

3. Review fixes (1ad4eea)

• Resolution order. run.sh now takes -r, then the shell's AOB_RUNTIME_IMAGE, then .env's, then the local default. Before, a value in .env was ignored.
• When it pulls. It pulls whenever the image isn't local, or the local copy came from that same repository. Docker Hub images are now refreshed too. A local build is never pulled over.
• Pinning. It pins the resolved image for the whole run under a tag only this process uses, removed on exit. A rebuild or pull of :dev mid-run no longer switches later trials' base. (FROM can't take a bare image id, which is why this uses a tag.)
• Resume guard. It records the image beside each job (<job>.runtime-image) and refuses to resume a job on a different image. Harbor's resume lock doesn't cover the base image.
• Exit status. It exits non-zero when a model's job couldn't start or resume. It also checks the cheap inputs before the image pull.
• build-runtime-image.sh: -tNAME and -t=NAME now count as naming a tag. The comments are corrected: rebuilding the same commit moves its tag, and old commit tags keep their images until you remove them.
• Publish note: publish-images.sh's note now points at the commit tag.

Docs

README, QUICKSTART and the note printed by publish-images.sh now say to export AOB_RUNTIME_IMAGE (or set it in .env) instead of retagging, and describe the commit tag. Tasks generated before this PR ignore the variable, so the docs say to regenerate them once.

What doesn't change

The image changes only the base. Each task still adds its own scenario_<id>/manifest.json, and shared/ still comes from overlays/private-data.yaml. For the open profile, shared/ is the repo copy inside the chosen image, so an image built from another commit brings that commit's open data.

Usage

bash benchmarks/harbor/run.sh -s <suite> -l <out> -r quay.io/assetopsbench/runtime:dev ...
bash benchmarks/harbor/run.sh -s <suite> -l <out> -r assetopsbench/runtime:3bb4f12 ...
AOB_RUNTIME_IMAGE=assetopsbench/runtime:3bb4f12 uv run harbor run ...

Caveats

• Old jobs can't be resumed. The template change alters every generated task, so Harbor won't resume a job started before this PR. run.sh already reports that and suggests moving the job aside.
• The quay.io image is arm64 only. quay.io/assetopsbench/runtime:dev is linux/arm64, so an amd64 host can't pull it. publish-images.sh builds both architectures.

Type of Change

• Infrastructure / Tooling Improvement

Industry Relevance

A published result can name the exact runtime build it ran on, and third parties can run a published image without retagging it.

Related Issues

• Refs: feat(harbor): run AssetOpsBench scenarios as Harbor tasks #558

Testing & Validation

• Oracle runs with AOB_RUNTIME_IMAGE set to a labelled copy of runtime:dev: the 3 open tasks, plus tsfm-1019 and fmsr-913 from the private suite with the private-data overlay. All 5 trials scored 1.000 with no errors.
• The variable is honoured: with AOB_RUNTIME_IMAGE=aob-test/does-not-exist:nope, the trial's build failed trying to pull exactly that name.
• What a trial sees: the tsfm-1019 task image built on the labelled base carries the label. With the overlay's mount it sees scenario_1019/manifest.json and a read-only shared/, and all five shared/… paths in its manifest resolve.
• run.sh image step: it pulls a quay.io reference and rejects a missing local tag. Registry detection gave the expected result for quay.io/…, icr.io/…, localhost:5000/…, localhost/…, assetopsbench/runtime:* and ubuntu:24.04.
• Commit tag: the build script with no arguments produced one image (bbb99529) tagged both assetopsbench/runtime:dev and assetopsbench/runtime:3bb4f12, and its /opt/aob/.aob-commit reads 3bb4f12409ee….
• run.sh end to end, with a one-scenario profile and a fake model id, so trials fail fast without calling a provider:
  • The first run printed the resolved id. The pin tag existed during the run and was gone after it, the job's .runtime-image record was written, and the trial built, ran and was scored.
  • A second run on the same image resumed the job, dropped the crashed trial, reran it, and exited 0.
  • A third run with -r set to a different image refused to resume, named the original image, and exited 1.
• Helpers: image_repo strips tags and digests. runtime:dev and :3bb4f12 count as local builds, while quay.io/… and couchdb:3.5 count as registry images. -r and the shell beat .env, which beats the default.
• bash -n passes on all three scripts.
• uv run pytest src/ -k "not integration": 714 passed, 19 failed. The base branch has the same 19 failures: the scorer and trace-exporter tests, and the iot invalid-site tests, which need a running CouchDB or no .env.
• Data integrity: no data or images are committed.

Checklist

• I have performed a self-review of my code.
• I have updated the documentation (README or /docs) accordingly.
• I have signed off my commits (DCO).