Fixes

.claude/skills/check-component/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: check-component
+description: >-
+  Verify a Viash component's change is fully deployed before (re)running a benchmark:
+  committed to origin/main, regenerated in build/main, and rebuilt as a fresh container
+  on ghcr (build_main tag). Use when a component edit doesn't seem to take effect on the
+  cluster / in a Nextflow or Seqera run, when a run errors with symptoms of an old image
+  (missing deps, wrong base image), or to confirm an image is not stale before launching.
+---
+
+# check-component
+
+A component change only reaches a benchmark run after **three** independent things
+happen. A failure at any stage means the cluster keeps using the **old container**, so
+the fix silently has no effect:
+
+1. **Source** — the change is committed and on `origin/main`.
+2. **Codegen** — CI's `Build` workflow regenerates the `build/main` deploy branch
+   (the inlined `target/nextflow/.../main.nf` that run scripts launch via
+   `--revision build/main`).
+3. **Container** — CI builds the component's Docker image and **pushes it to `ghcr.io`**
+   at the `build_main` tag. This is the step that most often lags or fails (heavy images,
+   source compiles, runner disk/time limits). Codegen redeploying does **not** imply the
+   container was rebuilt — they are separate steps.
+
+The decisive signal is the image's `org.opencontainers.image.revision` label: it records
+the git commit the container was built from. Comparing it to `origin/main` HEAD tells you
+whether the pushed image is current — read straight from the registry, without pulling the
+(multi-GB) image.
+
+## How to run
+
+```bash
+.claude/skills/check-component/check_component.sh <component>
+```
+
+Accepts a bare component name (`segger`), a `namespace/component` path
+(`methods_transcript_assignment/segger`), or a path to `config.vsh.yaml`. Optional 2nd arg
+overrides the image tag (default `build_main`; for a `build/<branch>` deploy the tag is
+`build_<branch>`).
+
+The script prints the three checks and a `VERDICT`. Report the verdict to the user and, when
+something is stale, which stage failed and what to do about it.
+
+## Interpreting results
+
+- **All three OK** → a run with `--pull-latest` will use the current build. Safe to launch.
+- **[1] source not on main** → the fix is uncommitted or on another branch (e.g. `fixes`).
+  Commit/merge to `main` first; the push to `main` is what triggers CI's `Build`.
+- **[2] build/main behind** → the `Build` workflow hasn't redeployed the latest `main` yet.
+  It usually runs within ~1 min of the merge; if not, check the Actions "Build" run.
+- **[3] container stale** (revision label ≠ `origin/main` HEAD) → the container push for the
+  latest commit hasn't succeeded. Either the `Build` run is still compiling the image, or the
+  image build **failed** (common for heavy GPU/compile images). Check the `Build` run for this
+  component; if it failed, rebuild + push the image manually:
+
+  ```bash
+  # from a linux/amd64 host with docker + `docker login ghcr.io`
+  viash ns build -q <component> --setup push
+  ```
+
+  (macOS/Apple-Silicon can't practically build amd64 CUDA images — use a Linux build host or
+  a build pod on the cluster.)
+
+- **[3] container `created-vs-commit: STALE`** (revision label **matches** HEAD, but the image
+  was **built before** the commit it is labelled with) → a **false-OK trap** the revision label
+  alone misses. The `revision` label was re-stamped onto old layers without a real rebuild —
+  Docker layer-cache reuse, a re-pushed/re-tagged manifest, or a heavy build that failed and
+  left the previous image in place. The deployed image does **not** contain the change even
+  though it claims the right SHA. Fix is the same as a plain stale container, but you must
+  **bust the cache** so the changed layers actually re-execute (build on a fresh builder, e.g. a
+  clean Docker-in-Docker pod, rather than one with warm cache). Tell: `image built` timestamp is
+  earlier than the revision commit's time.
+
+## Notes
+
+- Requires network access to the registry and `git fetch` access to `origin`. The script does
+  a `git fetch origin` itself.
+- The container check queries `ghcr.io` with an anonymous pull token, which works for public
+  packages. For a private package or a non-ghcr registry the container stage reports
+  `NOT FOUND`; verify via `docker manifest inspect` with credentials instead.
+- Project identity (organization, name, registry) is read from `_viash.yaml`; the image repo
+  path is `<registry>/<organization>/<name>/<namespace>/<component>`.

.claude/skills/check-component/check_component.sh

@@ -0,0 +1,169 @@
+#!/usr/bin/env bash
+# check_component.sh — verify a Viash component's change is fully deployed.
+#
+# Runs three checks and prints a verdict:
+#   [1/3] source: is the component's config committed to origin/main?
+#   [2/3] codegen: did build/main redeploy the latest origin/main commit?
+#   [3/3] container: was the ghcr :build_main image rebuilt from the latest commit?
+#
+# The container check reads the image's org.opencontainers.image.revision label
+# straight from ghcr (anonymous pull token) and compares it to origin/main HEAD,
+# so it does NOT pull the (multi-GB) image.
+#
+# Usage:
+#   check_component.sh <component>                         # e.g. segger
+#   check_component.sh <namespace>/<component>             # e.g. methods_transcript_assignment/segger
+#   check_component.sh <path/to/config.vsh.yaml>
+#   check_component.sh <component> <image-tag>             # tag defaults to build_main
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)" || { echo "ERROR: not in a git repo"; exit 2; }
+cd "$REPO_ROOT" || exit 2
+
+ARG="${1:-}"
+TAG="${2:-build_main}"
+[ -z "$ARG" ] && { echo "usage: check_component.sh <component|namespace/component|config-path> [image-tag]"; exit 2; }
+
+# ---- resolve the component config path ----
+if [[ "$ARG" == *config.vsh.yaml && -f "$ARG" ]]; then
+  CFG="$ARG"
+elif [[ -f "src/$ARG/config.vsh.yaml" ]]; then
+  CFG="src/$ARG/config.vsh.yaml"
+elif [[ -d "$ARG" && -f "$ARG/config.vsh.yaml" ]]; then
+  CFG="$ARG/config.vsh.yaml"
+else
+  HITS="$(find src -type d -name "$ARG" 2>/dev/null | while read -r d; do [ -f "$d/config.vsh.yaml" ] && echo "$d"; done)"
+  N="$(printf '%s\n' "$HITS" | grep -c . || true)"
+  if [ "$N" -eq 0 ]; then echo "ERROR: no component config found for '$ARG'"; exit 2; fi
+  if [ "$N" -gt 1 ]; then echo "ERROR: '$ARG' is ambiguous, matches:"; printf '%s\n' "$HITS"; exit 2; fi
+  CFG="$HITS/config.vsh.yaml"
+fi
+
+COMP_DIR="$(dirname "$CFG")"          # src/<namespace>/<component>
+COMP="$(basename "$COMP_DIR")"        # <component>
+NS="${COMP_DIR#src/}"; NS="${NS%/$COMP}"   # <namespace> (may be nested)
+
+# ---- project identity from _viash.yaml ----
+ident() { grep -E "$1" _viash.yaml | head -1 | sed -E 's/.*:[[:space:]]*//; s/[[:space:]]*$//'; }
+ORG="$(ident '^organization:')"
+PROJ="$(ident '^name:')"
+REG="$(ident 'docker_registry:')"; REG="${REG:-ghcr.io}"
+IMG="$ORG/$PROJ/$NS/$COMP"            # repo path after the registry
+
+echo "Component: $COMP  ($COMP_DIR)"
+echo "Image:     $REG/$IMG:$TAG"
+echo
+
+git fetch origin --quiet 2>/dev/null || echo "(warning: git fetch failed — results may be stale)"
+MAIN_SHA="$(git rev-parse origin/main 2>/dev/null || echo '')"
+MAIN_SUBJ="$(git log -1 --format=%s origin/main 2>/dev/null || echo '')"
+
+ok_src=1 ok_deploy=1 ok_cont=1
+
+# ================= [1/3] source committed to main =================
+echo "[1/3] Source committed to origin/main"
+echo "  origin/main HEAD:            ${MAIN_SHA:0:8}  $MAIN_SUBJ"
+UNCOMMITTED="$(git status --porcelain -- "$CFG" 2>/dev/null)"
+if [ -n "$UNCOMMITTED" ]; then
+  echo "  local uncommitted change:    YES  (working-tree edits to config not committed)"
+  ok_src=0
+else
+  echo "  local uncommitted change:    no"
+fi
+if git diff --quiet origin/main -- "$CFG" 2>/dev/null; then
+  echo "  config matches origin/main:  yes"
+else
+  echo "  config matches origin/main:  NO  (your version differs from what's on main)"
+  ok_src=0
+fi
+[ "$ok_src" -eq 1 ] && echo "  => OK: this component's config on origin/main matches your working tree" \
+                     || echo "  => NOT on main: merge/commit the change before it can build"
+echo
+
+# ================= [2/3] build/main regenerated =================
+echo "[2/3] build/main regenerated (codegen)"
+DEPLOY_MSG="$(git log -1 --format=%s origin/build/main 2>/dev/null || echo '')"
+DEPLOY_SHA="$(printf '%s' "$DEPLOY_MSG" | grep -oE '[0-9a-f]{40}' | head -1)"
+echo "  build/main last deploy:      ${DEPLOY_SHA:0:8}  ($DEPLOY_MSG)"
+if [ -n "$DEPLOY_SHA" ] && [ "$DEPLOY_SHA" = "$MAIN_SHA" ]; then
+  echo "  matches origin/main HEAD:    yes"
+  echo "  => OK: build/main codegen is current"
+else
+  echo "  matches origin/main HEAD:    NO"
+  echo "  => build/main has not redeployed the latest main yet (CI Build may be running)"
+  ok_deploy=0
+fi
+echo
+
+# ================= [3/3] container rebuilt on ghcr =================
+echo "[3/3] Container rebuilt on $REG ($TAG)"
+CONT_OUT="$(python3 "$SCRIPT_DIR/ghcr_inspect.py" "$REG" "$IMG" "$TAG" "$MAIN_SHA")"
+C_STATUS="$(printf '%s\n' "$CONT_OUT" | awk '/^STATUS/{print $2}')"
+if [ "$C_STATUS" != "ok" ]; then
+  C_DETAIL="$(printf '%s\n' "$CONT_OUT" | awk '/^DETAIL/{print $2}')"
+  echo "  image on registry:           NOT FOUND ($C_DETAIL)"
+  echo "  => no $TAG image has been pushed for this component"
+  ok_cont=0
+else
+  C_REV="$(printf '%s\n' "$CONT_OUT"  | awk '/^REVISION/{print $2}')"
+  C_CREATED="$(printf '%s\n' "$CONT_OUT" | awk '/^CREATED /{print $2}')"
+  C_CREATED_EPOCH="$(printf '%s\n' "$CONT_OUT" | awk '/^CREATED_EPOCH/{print $2}')"
+  C_BASE="$(printf '%s\n' "$CONT_OUT"  | awk '/^BASE_CREATED/{print $2}')"
+  C_LAYERS="$(printf '%s\n' "$CONT_OUT" | awk '/^LAYERS/{print $2}')"
+  C_MATCH="$(printf '%s\n' "$CONT_OUT" | awk '/^MATCH/{print $2}')"
+  echo "  image revision label:        ${C_REV:0:8}"
+  echo "  matches origin/main HEAD:    $([ "$C_MATCH" = yes ] && echo yes || echo 'NO  <-- STALE')"
+  echo "  image built:                 $C_CREATED"
+  echo "  base layer created:          $C_BASE"
+  echo "  layer count:                 $C_LAYERS"
+  if [ "$C_MATCH" = yes ]; then
+    echo "  => OK: container was built from the current origin/main commit"
+  else
+    echo "  => STALE: container was built from ${C_REV:0:8}, not ${MAIN_SHA:0:8}."
+    echo "     The container push for the latest commit has not succeeded"
+    echo "     (CI Build still running, or the image build failed)."
+    ok_cont=0
+  fi
+
+  # Cross-check: an honestly-built image is CREATED at (or after) the commit it
+  # claims to be built from. If 'created' predates the revision's commit time, the
+  # layers were NOT rebuilt for that revision — the label was re-stamped over old
+  # layers (Docker cache reuse, a re-pushed tag, or a heavy build that failed and
+  # left the previous image). The revision label alone can't catch this; the
+  # timestamp does. GRACE absorbs build-vs-git clock skew.
+  GRACE=120
+  REV_EPOCH="$(git show -s --format=%ct "$C_REV" 2>/dev/null)"
+  # Show BOTH times in UTC. The image 'created' is UTC (…Z) but git's default
+  # commit time carries the author's zone — comparing them as-is is a classic
+  # mistake (an image looks "older" than its commit only because of the offset).
+  REV_UTC="$(TZ=UTC git show -s --date=format-local:'%Y-%m-%dT%H:%M:%SZ' --format=%cd "$C_REV" 2>/dev/null)"
+  if [ -n "$REV_EPOCH" ] && printf '%s' "$C_CREATED_EPOCH" | grep -qE '^[0-9]+$'; then
+    echo "  revision commit time (UTC):  $REV_UTC"
+    if [ "$C_CREATED_EPOCH" -lt "$((REV_EPOCH - GRACE))" ]; then
+      echo "  created-vs-commit:           STALE  <-- image built BEFORE its own revision commit"
+      echo "  => STALE: image 'created' ($C_CREATED, UTC) predates the commit ${C_REV:0:8}"
+      echo "     it is labelled with ($REV_UTC). The revision label was stamped but the"
+      echo "     layers were NOT rebuilt (cache reuse / re-stamped tag / failed heavy build)."
+      echo "     Rebuild+push with a clean cache."
+      ok_cont=0
+    else
+      echo "  created-vs-commit:           ok (image built after its revision commit)"
+    fi
+  else
+    echo "  created-vs-commit:           (skipped — revision commit not in local history)"
+  fi
+fi
+echo
+
+# ================= verdict =================
+if [ "$ok_src" -eq 1 ] && [ "$ok_deploy" -eq 1 ] && [ "$ok_cont" -eq 1 ]; then
+  echo "VERDICT: OK — fully deployed. A run with --pull-latest will use the current build."
+  exit 0
+else
+  echo "VERDICT: NOT fully deployed — a run now would use a stale build. See the failing stage above."
+  [ "$ok_src"    -eq 0 ] && echo "  - source not on main: commit/merge the change to main"
+  [ "$ok_deploy" -eq 0 ] && echo "  - build/main behind: wait for / re-trigger the CI Build workflow"
+  [ "$ok_cont"   -eq 0 ] && echo "  - container stale: check the Build run for this component; if it failed, rebuild+push manually"
+  exit 1
+fi

.claude/skills/check-component/ghcr_inspect.py

@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+"""Inspect a container image's build provenance from a registry without pulling it.
+
+Usage: ghcr_inspect.py <registry> <image-repo-path> <tag> <expected-git-sha>
+
+Prints key/value lines consumed by check_component.sh:
+  STATUS ok|missing
+  DETAIL <code>            (only when missing)
+  REVISION <git-sha>       org.opencontainers.image.revision label
+  CREATED <iso8601>
+  CREATED_EPOCH <int>      CREATED as unix seconds (empty if unparseable)
+  BASE_CREATED <iso8601>   created time of the first (base) layer
+  LAYERS <n>
+  MATCH yes|no             REVISION == expected-git-sha
+"""
+import json
+import re
+import subprocess
+import sys
+from datetime import datetime, timezone
+
+
+def to_epoch(iso):
+    """ISO8601 (with optional fractional seconds and trailing Z) -> unix seconds.
+    Fractional seconds are dropped; registry timestamps are UTC. '' if unparseable."""
+    m = re.match(r"(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})", iso or "")
+    if not m:
+        return ""
+    y, mo, d, h, mi, s = (int(x) for x in m.groups())
+    return str(int(datetime(y, mo, d, h, mi, s, tzinfo=timezone.utc).timestamp()))
+
+ACCEPT = (
+    "application/vnd.oci.image.index.v1+json,"
+    "application/vnd.oci.image.manifest.v1+json,"
+    "application/vnd.docker.distribution.manifest.list.v2+json,"
+    "application/vnd.docker.distribution.manifest.v2+json"
+)
+
+
+def curl(url, accept=None, token=None, follow=False):
+    cmd = ["curl", "-s"]
+    if follow:
+        cmd.append("-L")
+    if token:
+        cmd += ["-H", f"Authorization: Bearer {token}"]
+    if accept:
+        cmd += ["-H", f"Accept: {accept}"]
+    cmd.append(url)
+    return subprocess.run(cmd, capture_output=True, text=True).stdout
+
+
+def jload(s):
+    try:
+        return json.loads(s)
+    except Exception:
+        return {}
+
+
+def main():
+    if len(sys.argv) < 5:
+        print("STATUS missing\nDETAIL bad-args")
+        return
+    reg, img, tag, main_sha = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]
+    base = "https://ghcr.io" if reg.strip() == "ghcr.io" else f"https://{reg}"
+
+    tok = jload(curl(f"{base}/token?scope=repository:{img}:pull&service={reg}")).get("token", "")
+    man = jload(curl(f"{base}/v2/{img}/manifests/{tag}", ACCEPT, tok))
+    if not man or man.get("errors"):
+        code = man.get("errors", [{}])[0].get("code") if man else "NO_RESPONSE"
+        print(f"STATUS missing\nDETAIL {code}")
+        return
+
+    if "manifests" in man:  # multi-arch index -> prefer linux/amd64
+        child = None
+        for m in man["manifests"]:
+            p = m.get("platform", {})
+            if p.get("os") == "linux" and p.get("architecture") == "amd64":
+                child = m["digest"]
+                break
+        child = child or man["manifests"][0]["digest"]
+        man = jload(curl(f"{base}/v2/{img}/manifests/{child}", ACCEPT, tok))
+
+    cfg_digest = (man.get("config") or {}).get("digest", "")
+    if not cfg_digest:
+        print("STATUS missing\nDETAIL no-config-digest")
+        return
+
+    cfg = jload(curl(f"{base}/v2/{img}/blobs/{cfg_digest}", token=tok, follow=True))
+    labels = (cfg.get("config") or {}).get("Labels") or {}
+    rev = labels.get("org.opencontainers.image.revision", "")
+    if not rev:  # fall back to scanning build history for the LABEL line
+        for h in cfg.get("history", []):
+            cb = h.get("created_by", "")
+            if "org.opencontainers.image.revision=" in cb:
+                rev = cb.split("org.opencontainers.image.revision=", 1)[1].split()[0].strip('"')
+                break
+
+    hist = cfg.get("history", [])
+    base_created = hist[0].get("created", "") if hist else ""
+    match = "yes" if (rev and main_sha and rev == main_sha) else "no"
+    created = cfg.get("created", "")
+    print("STATUS ok")
+    print(f"REVISION {rev}")
+    print(f"CREATED {created}")
+    print(f"CREATED_EPOCH {to_epoch(created)}")
+    print(f"BASE_CREATED {base_created}")
+    print(f"LAYERS {len(man.get('layers', []))}")
+    print(f"MATCH {match}")
+
+
+if __name__ == "__main__":
+    main()