Skip to content

fix(harbor): build the runtime image from a clean clone of HEAD - #587

Merged
ShuxinLin merged 2 commits into
feature/harbor-integrationfrom
fix/harbor-clean-runtime-image
Sep 30, 2026
Merged

ShuxinLin merged 2 commits into
feature/harbor-integrationfrom
fix/harbor-clean-runtime-image

Conversation

@ShuxinLin

Copy link
Copy Markdown
Collaborator

Description

base-image/Dockerfile does COPY . ., so building from the working tree baked in whatever was there. That included an untracked reports/results_table.csv with a ground_truth column for 56 private scenarios (8 of mini's 35, 41 of lite's 76). The file was in /opt/aob of assetopsbench/runtime:dev, where any agent running code in main could read it, and publish-images.sh would have pushed it to Docker Hub.

This PR applies two guards:

  1. Clean build. The new scripts/build-runtime-image.sh builds from a clone of HEAD that contains only that commit, so untracked and git-ignored files never enter the build.
  2. .dockerignore. It is the second guard, and the only one for a plain docker build ..

Fix Details

  • scripts/build-runtime-image.sh
    • Fetches HEAD at depth 1 into a temp directory, checks the clone is at the expected commit, builds with docker buildx build, and deletes the clone.
    • With no arguments it builds assetopsbench/runtime:dev --load. Any arguments go to buildx unchanged.
    • Warns when tracked files have uncommitted changes, because those are not built.
    • The clone keeps a real one-commit .git, so git rev-parse in StirrupAgent.get_version_command still works. A git worktree wouldn't work, because its .git is a pointer that doesn't resolve inside the container.
  • publish-images.sh builds the runtime image through the script. run.sh's error message, README, QUICKSTART, CODE-SANDBOX and the Dockerfile header now point at the script too.
  • .dockerignore
    • Uses **/.env / **/.env.*; the bare .env only matched at the top of the repo.
    • Leaves out the open scenarios' groundtruth*, rubric.json, reference_answer.json and scenario_meta.json. The verifier reads the copies under /tests, never these.
    • Leaves out reports/, logs/, src/tmp/, .claude/ and the kdd_tutorial folders.

Depends on #586 for the open profile. On this branch the generator still copies each scenario's groundtruth.txt into the task's build folder, so the per-task layer puts it back. #586 stops that. The two branches merge cleanly (checked with git merge-tree).

Impact on Benchmarking

  • No change to baselines: the image keeps the same code and dependencies. Only local files and answers are removed.
  • Baseline change

Related Issues

Verification Steps

  1. uv run pytest: not run. No Python changed.
  2. Manual verification:
    • Rebuilt assetopsbench/runtime:dev with the script from b7a7d84. It is 8.79 GB, down from 8.92 GB, and the temp clone was removed afterwards.

    • Old image vs new image:

      old image new image
      reports/results_table.csv present absent
      logs/, src/tmp/, .claude/, kdd_tutorial folders present absent
      Answer files in /opt/aob 3 0
      git rev-parse --short HEAD 0c45322 b7a7d84
      Commits in .git 1142 1
      .git size 157M 44M
    • harbor run --agent oracle on the open profile with the new image: 3 trials, 0 exceptions, reward 1.000. The verifier scores correctly without answers in the image.

    • With fix(harbor): run.sh mounts only the private suite's shared/ #586's generator and overlay, the tsfm-1019 task container on the new image has no answer-looking file anywhere, and the data load filled all 5 collections.

    • bash -n passes for all three scripts. publish-images.sh was not run (it pushes).

Checklist

  • I have added tests that prove my fix is effective.
  • My code follows the project's Ruff formatting and linting rules.
  • I have signed off my commits (DCO).

base-image/Dockerfile does `COPY . .`, so building from the working
tree baked in whatever sat there. An untracked reports/results_table.csv
with a ground_truth column for 56 private scenarios (8 of mini, 41 of
lite) reached assetopsbench/runtime:dev that way, readable by any agent
running code in `main`, and publish-images.sh would have pushed it.

scripts/build-runtime-image.sh now builds from a one-commit clone of
HEAD, so only committed files enter the context. The clone keeps a real
.git, which StirrupAgent.get_version_command needs for `git rev-parse`;
a git worktree's .git pointer would not resolve inside the container.
publish-images.sh, run.sh and the docs use the script.

.dockerignore stays as the second guard, and the only one for a plain
`docker build .`: env files at any depth (a bare `.env` matched only the
context root), the open scenarios' answer files, which the verifier
never reads from the image, and the local folders reports/, logs/,
src/tmp/, .claude/ and the kdd_tutorial work.

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Review of #587: the one-commit .git the clean clone kept still held
every committed blob, so `git show HEAD:.../groundtruth.txt` returned
the open scenarios' answers that .dockerignore had dropped from the
tree. Its reflog also recorded the builder's name and email, and its
index and reflog changed on every clone, so `COPY . .` never hit the
cache and each build re-downloaded the models.

build-runtime-image.sh now builds from `git archive HEAD` and passes
the commit as the AOB_COMMIT build arg. .dockerignore drops .git. A
`commit` stage in base-image/Dockerfile writes /opt/aob/.aob-commit,
which StirrupAgent.get_version_command now reads, and fails any build
without AOB_COMMIT, so a plain `docker build .` of the working tree
stops instead of baking in untracked files. The ARG lives only in that
stage, so a new commit invalidates the final layer and nothing else.

The script also keeps its default tag and --load when given other
flags (e.g. --no-cache), warns about untracked files it leaves out,
and checks for docker buildx up front.

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@ShuxinLin

Copy link
Copy Markdown
Collaborator Author

Second review pass, addressed in 47a806e:

  • No .git in the image. The one-commit .git still held every committed blob, so git show HEAD:…/groundtruth.txt returned the open scenarios' answers. Its reflog also recorded the builder's email, and its index made COPY . . miss the cache on every build. The script now builds from git archive HEAD and passes AOB_COMMIT; the Dockerfile writes it to /opt/aob/.aob-commit, and StirrupAgent.get_version_command reads that file.
  • A plain docker build now fails. Without AOB_COMMIT, a small commit stage fails the build in about a second with a pointer to the script. The ARG lives only in that stage, so a new commit invalidates just the final layer.
  • Script defaults: it keeps -t assetopsbench/runtime:dev and --load unless the caller passes a tag or an output. It also warns about untracked files it leaves out, and checks for buildx.

Verification:

  • .git absent; git show fails.
  • .aob-commit = 47a806e…, and the version command prints 47a806e.
  • No answer files in the image.
  • Server, evaluation and init_data modules import.
  • A second build of the same commit is fully cached (1 s).
  • Unit tests: the same 19 pre-existing failures as the base, none new.

Not changed:

  • The snap-Docker /tmp and sudo safe.directory edge cases.
  • Moving the model preload above COPY . ., which is a separate speed-up.

@ShuxinLin
ShuxinLin merged commit 326598d into feature/harbor-integration Sep 30, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant