feat(harbor): run AssetOpsBench scenarios as Harbor tasks

[DhavalRepo18]

Description

This PR does two things:

• It adds an opt-in way to run AssetOpsBench scenarios in parallel, as Harbor tasks with one task per scenario. Stirrup plugs in as a custom agent, and CouchDB runs as a per-trial Compose sidecar.
• It brings along fixes to the MCP servers and agent runners, a new rule for choosing the FMSR model, and changes to the TSFM model catalog.

Harbor gives each trial its own Compose project and names every container, network and volume after it, so each scenario gets its own CouchDB. That makes parallel scenario runs safe by design, rather than safe only if they're carefully sequenced.

Why this matters. The current path can't be run in parallel safely:

• scenario_suite_runner.py resets one shared CouchDB before each scenario, and loader._ensure_db drops each database before loading it.
• Run that loop concurrently and it deletes a live scenario's iot, workorder and vibration databases partway through its run.
• The agent then reads empty results, writes a plausible answer, and the trajectory scores as legitimate.
• Nothing crashes, so parallel numbers would be quietly wrong rather than visibly broken.

Existing behaviour doesn't change. scenario_suite_runner.py is untouched, and Harbor is an opt-in extra (uv sync --extra harbor).

Size: 73 files, +8,132 / −249. Most of that is uv.lock, about 41 MB of TTM weights under artifacts/, and the catalog scripts under benchmarks/harbor/scripts/. The PR targets aafeedback_changes, not main; see the reviewer notes.

What the code changes do

1. Harbor integration (the main feature), benchmarks/harbor/

• Task template (template/):
  • task.toml sets the trial's environment (COUCHDB_URL=http://couchdb:5984, SCENARIOS_DATA_DIR, AGENT_TRAJECTORY_DIR, …). Its healthcheck runs init_data.py <id>, so the scenario's data is loaded before Harbor starts the agent.
  • environment/docker-compose.yaml adds a CouchDB sidecar that publishes no host port. It also passes AOB_RUNTIME_IMAGE to the task's Dockerfile, so the base image is chosen per run, with no need to regenerate tasks (feat(harbor): pick the runtime image per run; tag builds by commit #588).
  • tests/test.sh runs the existing uv run evaluate, and tests/to_reward.py converts its report into Harbor's {reward, passed}.
• Task generator (adapter/generate_tasks.py) turns each scenario in a profile (open.yaml by default) into a Harbor task: the question, a thin Docker layer, the ground truth for the verifier, and an oracle solution that writes the known answer.
  • The task's image gets only manifest.json and the files that manifest names.
  • The answer files (groundtruth.txt, rubric.json, …) go only to tests/ and solution/, which Harbor uploads after the agent phase (fix(harbor): run.sh mounts only the private suite's shared/ #586).
• Runtime image (base-image/):
  • Python 3.12 with the repo, uv sync --no-dev --group otel --extra tsfm, and the 20 Hub models listed in models.txt (Chronos, Moirai, TimesFM, Toto, TSPulse, TTM) preloaded. UV_NO_SYNC=1 stops each trial's uv run from reinstalling the dev group.
  • Built for fast iteration (102646e). The models download in a stage of their own that depends only on models.txt and the preload script, and the uv cache lives in a BuildKit cache mount, out of the image. A source edit rebuilds in about 2 s, a pyproject.toml/uv.lock change in about 25 s, and a cold build takes about 7 minutes. The image is 6.5 GB, and each new commit adds only its ~60 MB of repo layers.
  • scripts/build-runtime-image.sh builds it from git archive HEAD, so untracked files and .git never reach it. A plain docker build fails fast and points to the script (fix(harbor): build the runtime image from a clean clone of HEAD #587). Each build is tagged assetopsbench/runtime:dev and assetopsbench/runtime:<commit> (feat(harbor): pick the runtime image per run; tag builds by commit #588).
  • scripts/publish-images.sh builds for both architectures and pushes :TAG, :<commit> and :latest.
• Stirrup as a Harbor agent (src/assetops_harbor/stirrup.py):
  • it runs stirrup-agent inside the task container, with Harbor's trial name as --run-id;
  • it maps Harbor --ak kwargs onto Stirrup's settings (code_enabled, code_backend, max_turns, temperature, reasoning_effort, workspace_dir);
  • it forwards the credential variables in CREDENTIAL_ENV_VARS, plus FMSR_MODEL_ID, into the agent phase only;
  • it loads .env from the working directory before checking credentials, and exported variables and --ae take precedence over it (feat(harbor): load .env for StirrupAgent credentials #575). When AOB_ENV_FILE is set, it reads that file instead, which is how run.sh makes ENV_FILE the only source (fix(harbor): harden run.sh jobs, retries, checks and inputs #591);
  • it stops before Harbor builds anything if the router credentials for the model or for FMSR_MODEL_ID are missing (fix(fmsr): take the FMSR model from FMSR_MODEL_ID, with no built-in default #583);
  • after the run it converts the saved record into Harbor's trajectory format (ATIF), so token counts show up;
  • it records the image's commit from /opt/aob/.aob-commit as the agent version (fix(harbor): build the runtime image from a clean clone of HEAD #587).
• Code-track sandbox (overlays/code-sandbox.yaml): an optional Docker-in-Docker sidecar, so code the agent writes can't see CouchDB's hostname or credentials. CODE-SANDBOX.md covers it (Doc for code execution #581).
• Private data (overlays/private-data.yaml), for the mini, lite and all profiles (feat(harbor): run mini/lite/all by mounting the private suite #577, fix(harbor): run.sh mounts only the private suite's shared/ #586). It mounts only the private suite's shared/, read-only. A wrong or relative AOB_PRIVATE_DIR fails the trial at start, instead of silently loading empty data.
• run.sh, the Harbor counterpart of benchmarks/run.sh. -s names the suite's scenarios_data, -p the profile and -r the runtime image.
  • Jobs (fix(harbor): harden run.sh jobs, retries, checks and inputs #591). It runs one Harbor job per profile, model and reasoning effort, at <leaderboard>/harbor-jobs/stirrup_agent__<profile>__<model>[__<effort>]. Re-running it resumes the job.
  • Task copies (fix(harbor): harden run.sh jobs, retries, checks and inputs #591). Each job gets its own copy of the tasks, generated once when it starts and kept in the gitignored benchmarks/harbor/datasets/jobs/. Later changes to the template, the suite or the generator no longer block a resume.
  • What a resume must match. Each job records the runtime image (feat(harbor): pick the runtime image per run; tag builds by commit #588), the code-sandbox tar and the suite (fix(harbor): harden run.sh jobs, retries, checks and inputs #591) it started on. A resume with a different image or suite is refused. A resume always loads the job's own code tar.
  • Runtime image (feat(harbor): pick the runtime image per run; tag builds by commit #588). A registry reference is pulled first; a local build is used as is. The image is pinned under a private tag for the whole run, so a :dev that moves mid-run can't switch later trials to a different base.
  • Code tar (fix(harbor): harden run.sh jobs, retries, checks and inputs #591). The code-sandbox image is built on every run (cached) and saved once per image id under ~/.cache/assetopsbench.
  • Checks before each model (fix(harbor): harden run.sh jobs, retries, checks and inputs #591). The model's router and FMSR_MODEL_ID's must answer and must accept their key. A model without a router prefix also needs FMSR_MODEL_ID.
  • Resume retries (fix(harbor): harden run.sh jobs, retries, checks and inputs #591). A resume reruns every trial that failed for a reason other than the model's own work: API or network errors, environment or verifier failures, Ctrl-C. test_run_sh.py keeps that list in step with Harbor.
  • Isolation (fix(harbor): harden run.sh jobs, retries, checks and inputs #591). A per-job lock stops two run.sh processes from working on one job. ENV_FILE is the only credential source. Relative paths resolve against your current directory.
  • Exit status (fix(harbor): harden run.sh jobs, retries, checks and inputs #591). The script exits non-zero when a model's job couldn't start or resume, or was skipped.
• metric.py: per-category averages over a finished job.
• Docs: README.md (how it works, with "Running a private profile end to end", "Running a public profile by hand" and "Running a private profile by hand"), QUICKSTART.md (the steps) and CODE-SANDBOX.md.
• .dockerignore excludes .env files at any depth, the open scenarios' answer files, local-only folders (reports/, logs/, src/tmp/, .claude/, the kdd_tutorial work), artifacts/output/ and .git (fix(harbor): build the runtime image from a clean clone of HEAD #587).

2. Clearer errors when a CouchDB database is missing (#554)

The iot, fmsr, wo and vibration servers now tell a missing database apart from a wrong key. Before, both produced errors like "unknown asset_id". Now a missing database returns "the data source does not exist in this environment; ... do not retry", without naming the database. Tests cover each server.

3. Agent runners pass the environment to MCP servers

Without an explicit env, the MCP SDK's stdio client passes on only HOME, PATH and a few other variables, so the servers never saw COUCHDB_URL and silently fell back to localhost:5984. The new mcp_server_env in src/agent/runner.py forwards the full environment, and the Stirrup, plan-execute, deep-agent and openai-agent runners now use it. This fixes a real bug and applies outside Harbor too. test_stirrup_mcp_env.py covers the Stirrup case.

4. FMSR model (#583)

• generate_failure_modes takes its model only from FMSR_MODEL_ID; there is no built-in default.
• Every agent runner sets it to the agent's own --model-id unless it is set explicitly (fmsr_env_overrides in src/agent/runner.py).
• Only the litellm_proxy/ and tokenrouter/ prefixes are accepted; watsonx support is dropped.
• The tool also works when the catalog hasn't been initialised.
• INSTRUCTIONS.md and docs/running_benchmark.md document the new behaviour.

5. TSFM model catalog and weights

• The repo's shared/tsfm/model_catalog.json drops ttm_96_28 and now lists six models: four local TTM checkpoints, the fine-tuned energy model ttm_energy_168_24, and ttm_1024_192_hub.
• The local checkpoints are committed under artifacts/tsfm_models/ as plain git files, not Git LFS (about 41 MB). artifacts/output/, for checkpoints an agent fine-tunes during a trial, is now ignored by git and Docker.
• Nine Python scripts under benchmarks/harbor/scripts/ generate, audit and fix the catalog, preload Hub models, fetch TTM checkpoints and smoke-test forecasts and tasks.
• src/servers/tsfm/tests/test_catalog_checkpoints.py checks that every local checkpoint in the catalog resolves, fits and forecasts.

6. Dependencies (pyproject.toml)

• tsfm moves from a dependency group to an optional extra and grows a lot: transformers[torch]>=5.3, gluonts, lightning, toto-models and others.
• A new harbor optional extra is added, and src/assetops_harbor is added to the packages.
• sktime goes up to >=1.2.0, and numba and pyod are added to the dev group.

Type of Change

• Infrastructure / Tooling Improvement

Industry Relevance

Third-party evaluators need concurrency and per-trial isolation before they can publish AssetOpsBench numbers, and this PR builds both into the runner. Each result also records the runtime image's commit, and a run can be pinned to one build with runtime:<commit>.

Related Issues

• Refs: experiment harbour design for AssetOpsBench #557, fix: distinct error message when a CouchDB database is missing #554, feat(harbor): load .env for StirrupAgent credentials #575, feat(harbor): run mini/lite/all by mounting the private suite #577, Doc for code execution #581, fix(fmsr): take the FMSR model from FMSR_MODEL_ID, with no built-in default #583, fix(harbor): run.sh mounts only the private suite's shared/ #586, fix(harbor): build the runtime image from a clean clone of HEAD #587, feat(harbor): pick the runtime image per run; tag builds by commit #588, docs(harbor): correct stale docs, docstrings and comments #590, fix(harbor): harden run.sh jobs, retries, checks and inputs #591

Running the benchmark

Run every command from the repo root. You need Docker running (at least 4 GB of memory) and model credentials in .env (LITELLM_* and/or TOKENROUTER_*). Use a litellm_proxy/ or tokenrouter/ model, or set FMSR_MODEL_ID to one: FMSR's generate_failure_modes accepts only those two routers. benchmarks/harbor/QUICKSTART.md has the step-by-step version and common problems.

End to end (as of ba120aa)

# 0. Once: install the Harbor extra.
uv sync --dev --extra harbor

# 1. Build the runtime image from HEAD, and pin the run to this build's commit tag.
bash benchmarks/harbor/scripts/build-runtime-image.sh
RUNTIME=assetopsbench/runtime:$(git rev-parse HEAD | cut -c1-7)

# 2. Stirrup on the private mini suite, with the Docker code sandbox.
#    Rerun the same command to resume.
bash benchmarks/harbor/run.sh \
  -s <path-to>/AssetOpsBenchScenarioGeneration/scenarios_data \
  -l ~/AssetOpsBenchRuns/leaderboard \
  -p benchmarks/scenario_suite/mini.yaml \
  -r "$RUNTIME" -n 4 \
  -m "litellm_proxy/azure/gpt-5.6-sol max"

# 3. Results: Harbor's viewer, and per-category means for one job.
uv run harbor view ~/AssetOpsBenchRuns/leaderboard/harbor-jobs
uv run python benchmarks/harbor/metric.py --job-dir \
  ~/AssetOpsBenchRuns/leaderboard/harbor-jobs/stirrup_agent__mini__litellm_proxy-azure-gpt-5.6-sol__max

• The -s folder: run.sh wires the private suite from it. It mounts the suite's shared/ read-only into each trial and generates each job's tasks from it.
• Defaults: leave out -m to run the 8 default models, and -p to run the all profile. Each trial runs a privileged dind sidecar, so keep -n around 4 on a laptop-sized Docker VM.
• Private catalog first: its ttm_energy_168_24 must point at artifacts/tsfm_models/ before you run on a new image:
  uv run python benchmarks/harbor/scripts/apply_catalog_fixes.py --catalog <path-to>/scenarios_data/shared/catalog/model_catalog.json --move-energy --write
• Not the published image yet: quay.io/assetopsbench/runtime:dev was built before the model move and the --no-dev change, so build locally as in step 1.
• Disk: the first run saves the code-sandbox image (about 450 MB) to ~/.cache/assetopsbench/.

By hand, without run.sh

The same suites with plain Harbor commands. Both run Stirrup with the Docker code sandbox.

• Credentials come from .env, which StirrupAgent loads itself. It forwards them into the agent phase only, and fails at once if its model's router pair is missing. --ae KEY=VALUE overrides them for a single run.
• AOB_RUNTIME_IMAGE is the runtime image the tasks build FROM. Set it to the $RUNTIME from step 1; unset, it falls back to the local assetopsbench/runtime:dev.
• AOB_CODE_TAR is the code-sandbox image each trial loads. run.sh keeps its own copy under ~/.cache/assetopsbench/, so for a run by hand, save it once:

docker build -t assetops-code:dev \
  -f src/agent/stirrup_agent/Dockerfile.code src/agent/stirrup_agent
docker save assetops-code:dev -o ~/assetops-code.tar

Open suite. There is no --scenario-root, so the tasks use the repo's own scenario data (src/couchdb/scenarios_data):

export AOB_RUNTIME_IMAGE=$RUNTIME AOB_CODE_TAR=~/assetops-code.tar

uv run python benchmarks/harbor/adapter/generate_tasks.py \
  --profile benchmarks/scenario_suite/open.yaml \
  --output-dir benchmarks/harbor/datasets/assetopsbench-open \
  --dataset-name assetopsbench/open \
  --overwrite && \
uv run harbor run -y \
  -p benchmarks/harbor/datasets/assetopsbench-open \
  --extra-docker-compose benchmarks/harbor/overlays/code-sandbox.yaml \
  --agent assetops_harbor.stirrup:StirrupAgent \
  --model litellm_proxy/azure/gpt-5.6-sol \
  --ak code_enabled=true --ak code_backend=docker --ak allow_docker_backend=true \
  --ak workspace_dir=/workspace-share \
  --n-concurrent 4

Mini suite. --scenario-root points at the private suite, and overlays/private-data.yaml mounts its shared/ from AOB_PRIVATE_DIR. Give an absolute path: Compose would resolve a relative one against the task's folder, and the trial would fail at start.

export AOB_PRIVATE_DIR=<path-to>/AssetOpsBenchScenarioGeneration/scenarios_data
export AOB_RUNTIME_IMAGE=$RUNTIME AOB_CODE_TAR=~/assetops-code.tar

uv run python benchmarks/harbor/adapter/generate_tasks.py \
  --scenario-root "$AOB_PRIVATE_DIR" \
  --profile benchmarks/scenario_suite/mini.yaml \
  --output-dir benchmarks/harbor/datasets/assetopsbench-mini \
  --dataset-name assetopsbench/mini \
  --overwrite && \
uv run harbor run -y \
  -p benchmarks/harbor/datasets/assetopsbench-mini \
  --extra-docker-compose benchmarks/harbor/overlays/private-data.yaml \
  --extra-docker-compose benchmarks/harbor/overlays/code-sandbox.yaml \
  --agent assetops_harbor.stirrup:StirrupAgent \
  --model litellm_proxy/azure/gpt-5.6-sol \
  --ak code_enabled=true --ak code_backend=docker --ak allow_docker_backend=true \
  --ak workspace_dir=/workspace-share \
  --n-concurrent 4

Each private task's container sees only its own scenario_<id>/manifest.json and the suite's shared/, mounted read-only. It sees no other scenario's folder and no answer files.

• Check before spending tokens: run the open suite's harbor run with --agent oracle in place of the --extra-docker-compose, --agent, --model and --ak flags. Expect 3 trials, 0 exceptions and reward 1.000; anything less is a setup problem, not a model problem.
• Tools-only track: drop the code-sandbox overlay and the --ak flags, and pass --ak code_enabled=false. Avoid code_backend=local: agent code then runs next to CouchDB and the forwarded credentials, and could bypass the MCP tools the benchmark measures.
• Subset: mini is 35 tasks. Add -i 'fmsr-*' (or another category) to the harbor run part for a quick subset.
• Results land in jobs/<timestamp>/<task>__<id>/: result.json has the rewards and token totals, agent/trajectory.json the trajectory in ATIF form, and verifier/ the reward and evaluation logs. uv run harbor view jobs opens them in Harbor's viewer.
• Resume: with the same exports, run uv run harbor jobs resume -p jobs/<timestamp>. Harbor refuses if the tasks changed since the job started, for example after regenerating from an edited template or suite.

Testing & Validation

• Unit tests on this branch's head (ba120aa):

  uv run pytest src/ -k "not integration" gives 718 passed, 19 failed and 3 skipped. None of the 19 is caused by this PR:

  • 8 also fail on aafeedback_changes: evaluation/tests/test_static_json_scorer.py (6) and observability/tests/test_file_exporter.py (2).
  • 11 are the existing test_invalid_site tests in iot/tests/test_tools.py. This PR adds a test class to that file but doesn't change these tests. They fail with Connection refused on localhost:5984 when a local .env sets CouchDB variables and no CouchDB is running. Run them without .env, or with CouchDB up.

  test_catalog_checkpoints.py (12 passed) loads, fits and forecasts every local checkpoint at its catalog path, ttm_energy_168_24 included.

• Scenario validation: Harbor's oracle agent (which writes the known answer) scored 1.000 with no exceptions:

  • on the 3 open tasks;
  • on 13 private tasks with overlays/private-data.yaml;
  • on 5 tasks (3 open, 2 private) with AOB_RUNTIME_IMAGE pointing at a different image;
  • on the 3 open tasks again with the current runtime image, runtime:102646e (ba120aa changes only the README).

  A 100% oracle pass is Harbor's own gate that a task can be scored at all.

• Agent run: Stirrup with litellm_proxy/azure/gpt-5.6-sol, the local code backend and the open suite gave 3 trials, 0 exceptions and a mean reward of 0.952.

  • wosr-1 and wosr-2 scored 1.0.
  • wosr-3 scored 0.857. It matched 6 of 7 keys, and gave the end of the queried window rather than the last observation's timestamp.
  • No trajectory referenced groundtruth, /tests or /solution.
  • Mini with -i 'fmsr-*' and the Docker sandbox, set up as under "By hand", gave 5 trials, 0 exceptions, 3/5 passed and a mean reward of 0.760. The catalog, IoT and FMSR tools returned data in every trial.
  • fmsr-913 with feat(harbor): run mini/lite/all by mounting the private suite #577 (local code backend): the data loaded and the tools returned no errors.

• Not yet run end to end: the agent runs above, and the private and alternate-image oracle runs, predate docs(harbor): correct stale docs, docstrings and comments #590, fix(harbor): harden run.sh jobs, retries, checks and inputs #591, the energy model's move and the runtime image changes in 2970b5d and 102646e. run.sh's changes in fix(harbor): harden run.sh jobs, retries, checks and inputs #591 were tested with Harbor and Docker stubbed (see fix(harbor): harden run.sh jobs, retries, checks and inputs #591). The open suite with the Docker sandbox, as written under "By hand", has not been run.

• Per-trial isolation, observed during a concurrent run:

  uv run harbor run -y -p benchmarks/harbor/datasets/assetopsbench-open --agent oracle --n-concurrent 2 &
  sleep 20 && docker ps --format '{{.Names}}\t{{.Ports}}' | grep couchdb

  This shows two rows with different project prefixes and no host ports.

• No answers in the images:

  • Searching the whole filesystem of the current runtime image (runtime:102646e) finds no groundtruth*, rubric.json, reference_answer*, scenario_meta.json or .env files and no .git. None of the local-only folders .dockerignore excludes (reports/, logs/, src/tmp/, .claude/, artifacts/output/, jobs/, generated tasks) is present.
  • All 231 all-profile tasks carry only manifest.json in their image.

• Data integrity: generated tasks and run output are gitignored, and no scenario data is duplicated into the repo.

Reviewer notes

• Base branch: aafeedback_changes is already fully merged into main. The only commits on main that it lacks are fix: distinct error message when a CouchDB database is missing #554's, and this branch already contains them. Retargeting to main would leave the same change, minus fix: distinct error message when a CouchDB database is missing #554's files.

• Committed model weights: about 41 MB of safetensors, all under artifacts/tsfm_models/, as plain git files rather than Git LFS. The largest file is 20 MB.

• ttm_energy_168_24 moved. It sat in artifacts/output/tuned_models/, which artifacts/README.md reserves for agent output that is never committed. It now ships from artifacts/tsfm_models/, and the repo's catalog points there.

  A catalog outside the repo that still names the old path fails for this model once the runtime image is rebuilt. The private suite's shared/catalog/model_catalog.json does. Update it with the apply_catalog_fixes.py command under "End to end" before the next private run on a new image.

• Ground truth in the image: fixed.

  • Tasks copy only manifest.json and the files it names into their image (fix(harbor): run.sh mounts only the private suite's shared/ #586).
  • The runtime image is built from git archive HEAD, with the open scenarios' answer files and .git excluded (fix(harbor): build the runtime image from a clean clone of HEAD #587).
  • The private mount exposes only shared/ (fix(harbor): run.sh mounts only the private suite's shared/ #586).
  • run.sh's per-job task copies stay in the repo's gitignored datasets/, not beside the results (fix(harbor): harden run.sh jobs, retries, checks and inputs #591).

• Fixed since the first review:

  • resume reran only one error type (fix(harbor): harden run.sh jobs, retries, checks and inputs #591);
  • the MCP env test deleted a pre-set COUCHDB_URL;
  • stale docs, docstrings and template comments (docs(harbor): correct stale docs, docstrings and comments #590, 038d900 and 2daca75, which also cut the Harbor docs down to what a reader needs now);
  • duplicate dependencies in pyproject.toml and a duplicate .gitignore rule;
  • the energy model's location;
  • the code-sandbox loader's message when no code tar is given named no image (581a538);
  • the runtime image carried the dev group (pytest and the Jupyter stack), and a timed-out download from it failed a build (2970b5d);
  • every rebuild downloaded the 3.9 GB of Hub models again, and the image kept a 2.2 GB uv cache beside its venv (102646e). A source edit now rebuilds in seconds, and the image dropped from 8.75 GB to 6.5 GB.

• Known issues, not fixed in this PR, most severe first:

  1. stirrup.py forwards every provider credential it finds (AWS, Anthropic, OpenAI, Gemini, watsonx) into the agent container, whatever router the model uses. Under code_backend=local, agent code can read them. run.sh uses the Docker sandbox, whose code containers don't get them.
  2. The verifier's environment in template/task.toml has no TOKENROUTER_* variables. A tokenrouter/ judge therefore can't authenticate, and every llm_judge scenario scores 0 without an error.
  3. Under code_backend=local, the agent is root in the container the verifier runs in. It could write a run record to /logs/agent or edit /opt/aob. The Docker sandbox, which run.sh uses, isn't affected.
  4. plan-execute and stirrup-agent still default to a watsonx model, which FMSR rejects, so a run with no --model-id has no FMSR generate tools. run.sh refuses such a model unless FMSR_MODEL_ID names a router model, and the docs no longer use one (docs(harbor): correct stale docs, docstrings and comments #590). The command-line tools are unchanged.
  5. run.sh edges: two processes taking over a dead job's lock in the same instant could both proceed; Harbor still loads a .env.local from the repo root, if there is one; old code tars and task copies are never cleaned up.

[ShuxinLin]

What the code changes do

PR #558 does two things. It adds an opt-in way to run AssetOpsBench scenarios in parallel using Harbor, where each trial gets its own CouchDB. It also brings along a batch of fixes to the servers and changes to the TSFM model catalog. It's large: 57 files, +7,004 / −172, much of it uv.lock and model weights. It targets aafeedback_changes, not main.

1. Harbor integration (the main feature)

The reason for it: the current runner resets one shared CouchDB before every scenario, so running scenarios at once would wipe a live scenario's data. The agent would then get empty results and still produce a plausible-looking answer that gets scored. Harbor starts a separate Compose project per trial, so each scenario gets its own database.

• Task generator (benchmarks/harbor/adapter/generate_tasks.py): turns each scenario in open.yaml into a Harbor task. That task contains the question, a thin Docker layer, the ground-truth files for the verifier, and an "oracle" solution that writes the correct answer.
• Task template (benchmarks/harbor/template/):
  • task.toml sets the environment variables (COUCHDB_URL=http://couchdb:5984, …) and loads the scenario's data through a healthcheck running init_data.py <id>.
  • docker-compose.yaml adds a CouchDB sidecar that doesn't publish any host port.
  • test.sh runs the existing uv run evaluate, and to_reward.py converts its report into Harbor's {reward, passed}.
• Base image (base-image/Dockerfile): Python 3.12 with the repo, uv sync --extra tsfm --group otel, and the Hugging Face models listed in models.txt pre-downloaded.
• Stirrup as a Harbor agent (src/assetops_harbor/stirrup.py): runs uv run stirrup-agent inside the task container. It forwards only an allow-list of credential variables, uses Harbor's trial name as --run-id, and stops before anything is built if the router credentials are missing. After the run it converts the saved record into Harbor's trajectory format (ATIF) so token counts show up.
• Code-track sandbox (overlays/code-sandbox.yaml): an optional Docker-in-Docker sidecar, so code the agent writes can't see CouchDB's hostname or credentials.
• metric.py: per-category averages over a finished job.

2. Clearer errors when a CouchDB database is missing (from #554)

The iot, fmsr, wo and vibration servers now tell apart a missing database and a wrong key. Before, both produced errors like "unknown asset_id". Now a missing database returns "the data source does not exist … do not retry with other arguments", without naming the database. Tests cover this.

3. Stirrup runner passes the environment to MCP servers

src/agent/stirrup_agent/runner.py now sets "env": dict(os.environ). Without it, the MCP SDK passed on only HOME, PATH and a few other variables, so the servers silently fell back to localhost:5984. This fixes a real bug and applies outside Harbor too.

4. TSFM model catalog and weights

• model_catalog.json is rewritten: the ttm_96_28 entry is removed and entries are added for several local TTM checkpoints, a fine-tuned energy model (ttm_energy_168_24), and many Hugging Face models (Chronos, Moirai, TimesFM, Toto, TSPulse, …).
• About 41 MB of .safetensors files are committed under artifacts/ as plain git blobs, not Git LFS.
• Nine helper scripts are added under benchmarks/harbor/scripts/ (catalog audit and fixes, preloading, smoke tests), plus src/servers/tsfm/tests/test_catalog_checkpoints.py.

5. Dependencies (pyproject.toml)

• tsfm moves from a dependency group to an extra and grows a lot: transformers[torch]>=5.3, gluonts, lightning, toto-models, and others.
• A new harbor extra is added, and sktime goes up to >=1.2.0.