Fix/rebase image and add scan gate

[l-mansouri]

Overview

This PR implements the necessary changes for cloudos-cli container to pass the harbor scan without any critical or high vulnerabilities.

Changes

• Refactors Dockerfile and release.yml to account for the new container

Acceptance Criteria

The container was used in this run of CVA pipeline

https://cloudos.lifebit.ai/app/advanced-analytics/analyses/6aa17084ad77e94ee9c9fa92

testing the container

Using a custom script I also tested the container

custom script

#!/usr/bin/env bash
#
# Functional tests for the cloudos-cli Docker image. Self-contained: run it
# from anywhere, no checkout required.
#
#   ./test_docker_image.sh                                  # tests v2.96.0-test
#   ./test_docker_image.sh quay.io/lifebitaiorg/cloudos-cli:v2.95.1
#
# ci.yml already covers CLI behaviour on the host, so these tests cover only
# what is specific to the container: entry points resolve, the interpreter and
# its dependencies import, TLS and CA certificates work, `ps` exists for
# Nextflow, and the image behaves the way Nextflow invokes it.
#
# Tier 1 needs no credentials. Tier 2 hits the live API with read-only
# commands — it creates, modifies and deletes nothing — and runs when
# credentials are supplied any of these ways:
#
#   CLOUDOS_TOKEN=... CLOUDOS_WORKSPACE_ID=... ./test_docker_image.sh
#   ./test_docker_image.sh --apikey <token> --workspace-id <id>
#   echo 'CLOUDOS_TOKEN=...'       >  ~/.cloudos-test.env
#   echo 'CLOUDOS_WORKSPACE_ID=...' >> ~/.cloudos-test.env   # auto-sourced
#
# Prefer the env-var or dotenv forms: a token passed as --apikey is visible in
# your shell history and in `ps` output while the test runs.
#
# Tier 3 tests a named profile, mounting your host $HOME/.cloudos into the
# container so it can read the profile:
#
#   ./test_docker_image.sh --profile my-profile --project-name my-project
#
# Options:
#   --apikey <token>        API key            (env CLOUDOS_TOKEN)
#   --workspace-id <id>     workspace id       (env CLOUDOS_WORKSPACE_ID)
#   --cloudos-url <url>     platform url       (env CLOUDOS_URL)
#   --project-name <name>   required by `datasets ls` (env CLOUDOS_PROJECT_NAME)
#   --profile <name>        profile to exercise in tier 3 (env CLOUDOS_PROFILE)
#   --config-dir <path>     profile dir to mount, default ~/.cloudos
#   --expect-version <v>    override the expected `cloudos --version`
#
# Exits 0 if every test passed, 1 if any failed, 2 if the image is unusable.

set -uo pipefail

IMAGE_DEFAULT="quay.io/lifebitaiorg/cloudos-cli:v2.96.0-test"
IMAGE=""
CLOUDOS_URL="${CLOUDOS_URL:-https://cloudos.lifebit.ai}"
EXPECT_VERSION="${EXPECT_VERSION:-}"
# `datasets ls` requires --project-name; the dataset tests are skipped without it.
PROJECT_NAME="${CLOUDOS_PROJECT_NAME:-}"
# Profile to exercise. Profiles live in $HOME/.cloudos on the host, so the
# directory is mounted into the container for these tests.
PROFILE="${CLOUDOS_PROFILE:-}"
CONFIG_DIR="${CLOUDOS_CONFIG_DIR:-${HOME}/.cloudos}"

while [[ $# -gt 0 ]]; do
  case "$1" in
    --apikey)        CLOUDOS_TOKEN="$2"; shift 2 ;;
    --workspace-id)  CLOUDOS_WORKSPACE_ID="$2"; shift 2 ;;
    --cloudos-url)   CLOUDOS_URL="$2"; shift 2 ;;
    --project-name)  PROJECT_NAME="$2"; shift 2 ;;
    --profile)       PROFILE="$2"; shift 2 ;;
    --config-dir)    CONFIG_DIR="$2"; shift 2 ;;
    --expect-version) EXPECT_VERSION="$2"; shift 2 ;;
    -h|--help)       sed -n '2,34p' "$0" | sed 's/^# \?//'; exit 0 ;;
    -*)              echo "unknown option: $1" >&2; exit 2 ;;
    *)               IMAGE="$1"; shift ;;
  esac
done
IMAGE="${IMAGE:-${IMAGE_ENV:-$IMAGE_DEFAULT}}"

# Credentials from a dotenv file, if present and not already in the environment.
if [[ -f "${HOME}/.cloudos-test.env" ]]; then
  # shellcheck disable=SC1091
  set -a; . "${HOME}/.cloudos-test.env"; set +a
fi

PASS=0; FAIL=0
declare -a FAILED=()

run() {
  local desc="$1"; shift
  [[ "${1:-}" == "--" ]] && shift
  local out rc
  out="$("$@" 2>&1)"; rc=$?
  if [[ $rc -eq 0 ]]; then
    printf '  \033[32mPASS\033[0m  %s\n' "$desc"; PASS=$((PASS+1))
  else
    printf '  \033[31mFAIL\033[0m  %s (exit %s)\n' "$desc" "$rc"
    printf '        %s\n' "${out:0:400}"
    FAIL=$((FAIL+1)); FAILED+=("$desc")
  fi
}

expect() {
  local desc="$1" pattern="$2"; shift 2
  [[ "${1:-}" == "--" ]] && shift
  local out rc
  out="$("$@" 2>&1)"; rc=$?
  if [[ $rc -eq 0 && "$out" =~ $pattern ]]; then
    printf '  \033[32mPASS\033[0m  %s\n' "$desc"; PASS=$((PASS+1))
  else
    printf '  \033[31mFAIL\033[0m  %s (exit %s, expected match: %s)\n' "$desc" "$rc" "$pattern"
    printf '        %s\n' "${out:0:400}"
    FAIL=$((FAIL+1)); FAILED+=("$desc")
  fi
}

din() { docker run --rm "$IMAGE" "$@"; }

command -v docker >/dev/null || { echo "docker not found" >&2; exit 2; }
echo "image: $IMAGE"
if ! docker image inspect "$IMAGE" >/dev/null 2>&1; then
  echo "pulling $IMAGE ..."
  docker pull -q "$IMAGE" >/dev/null || { echo "cannot pull $IMAGE" >&2; exit 2; }
fi
echo "digest: $(docker inspect "$IMAGE" --format '{{if .RepoDigests}}{{index .RepoDigests 0}}{{else}}<local build>{{end}}')"

# Expected version: an explicit --expect-version, else the source tree if this
# script is inside a checkout, else derived from the image tag (v2.96.0-test ->
# 2.96.0).
if [[ -z "$EXPECT_VERSION" ]]; then
  src="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." 2>/dev/null && pwd)/cloudos_cli/_version.py"
  if [[ -f "$src" ]]; then
    EXPECT_VERSION="$(grep -Po "__version__\s*=\s*['\"]\K[^'\"]+" "$src")"
  else
    EXPECT_VERSION="$(printf '%s' "${IMAGE##*:}" | sed -e 's/^v//' -e 's/-.*$//')"
  fi
fi

echo
echo "== Tier 1: image integrity (no credentials) =="

expect "cloudos --version reports $EXPECT_VERSION" "$EXPECT_VERSION" -- din cloudos --version
run "cloudos --help exits 0" -- din cloudos --help

for group in job workflow project queue datasets bash configure interactive-session; do
  run "command group resolves: cloudos $group --help" -- din cloudos "$group" --help
done

run "python imports cloudos_cli and its runtime deps" -- \
  din python -c "import cloudos_cli, pandas, numpy, requests, click, rich_click"

# Nextflow shells out to `ps` for task tracing; without procps every task in a
# CloudOS run loses its metrics.
run "procps present (ps works) - required by Nextflow" -- din ps -p 1

# API calls are HTTPS: a container without CA certificates fails only at
# runtime, against the real platform.
run "TLS/CA certificates work against $CLOUDOS_URL" -- din python -c "
import requests
r = requests.get('$CLOUDOS_URL', timeout=30)
assert r.status_code < 500, r.status_code
"

# Nextflow invokes containers as: docker run <image> /bin/bash -c '<script>'.
# An inherited ENTRYPOINT (miniforge3 ships tini) or a missing shell breaks
# that shape specifically.
expect "runs under 'bash -c' the way Nextflow invokes it" "$EXPECT_VERSION" -- \
  din /bin/bash -c 'cloudos --version'

expect "no ENTRYPOINT set (would prefix every Nextflow command)" '^$' -- \
  docker inspect "$IMAGE" --format '{{if .Config.Entrypoint}}{{json .Config.Entrypoint}}{{else}}{{end}}'

run "cloudos_cli module files are executable" -- \
  din /bin/bash -c 'test -x /cloudos_cli/__main__.py'

run "writable HOME in container (cloudos configure needs it)" -- \
  din /bin/bash -c 'touch "$HOME/.write-probe" && rm "$HOME/.write-probe"'

echo
echo "== Tier 2: live API, read-only =="

if [[ -z "${CLOUDOS_TOKEN:-}" || -z "${CLOUDOS_WORKSPACE_ID:-}" ]]; then
  cat <<'EOS'
  SKIP  no credentials. Supply them with either:
          CLOUDOS_TOKEN=... CLOUDOS_WORKSPACE_ID=... <this script>
          <this script> --apikey <token> --workspace-id <id>
          ~/.cloudos-test.env  containing CLOUDOS_TOKEN= and CLOUDOS_WORKSPACE_ID=
EOS
else
  common=(--cloudos-url "$CLOUDOS_URL" --apikey "$CLOUDOS_TOKEN" --workspace-id "$CLOUDOS_WORKSPACE_ID")
  # Mirrors ci.yml's job_list / workflow_list / project_list / queue_list /
  # dataset_list jobs, exercised through the container.
  run "cloudos job list"      -- din cloudos job list "${common[@]}"
  run "cloudos workflow list" -- din cloudos workflow list "${common[@]}"
  run "cloudos project list"  -- din cloudos project list "${common[@]}"
  run "cloudos queue list"    -- din cloudos queue list "${common[@]}"

  # datasets ls needs a project. ci.yml passes --project-name too.
  if [[ -n "$PROJECT_NAME" ]]; then
    run "cloudos datasets ls --project-name '$PROJECT_NAME'" -- \
      din cloudos datasets ls "${common[@]}" --project-name "$PROJECT_NAME"
  else
    echo "  SKIP  cloudos datasets ls — needs --project-name <name>"
  fi
fi

echo
echo "== Tier 3: named profile =="

# Profiles are read from $HOME/.cloudos inside the container, which a fresh
# container does not have — so the host's config directory is mounted
# read-only at /root/.cloudos (the image runs as root, HOME=/root). This is
# also the shape to use in production if you want the container to pick up a
# profile rather than take --apikey on the command line.
if [[ -z "$PROFILE" ]]; then
  echo "  SKIP  no --profile given"
elif [[ ! -d "$CONFIG_DIR" ]]; then
  echo "  SKIP  --profile '$PROFILE' given but config dir not found: $CONFIG_DIR"
  echo "        create it with: cloudos configure --profile $PROFILE"
else
  dinp() { docker run --rm -v "${CONFIG_DIR}:/root/.cloudos:ro" "$IMAGE" "$@"; }

  expect "profile '$PROFILE' is visible inside the container" "$PROFILE" -- \
    dinp cloudos configure list-profiles

  # With a profile, apikey/url/workspace-id come from the config file, so they
  # are deliberately not passed here — that is what makes this a profile test.
  if [[ -n "$PROJECT_NAME" ]]; then
    run "cloudos datasets ls --profile '$PROFILE' --project-name '$PROJECT_NAME'" -- \
      dinp cloudos datasets ls --profile "$PROFILE" --project-name "$PROJECT_NAME"
  else
    run "cloudos datasets ls --profile '$PROFILE'" -- \
      dinp cloudos datasets ls --profile "$PROFILE"
  fi

  run "cloudos job list --profile '$PROFILE'" -- dinp cloudos job list --profile "$PROFILE"
fi

echo
echo "================ summary ================"
printf 'image: %s\npass=%d  fail=%d\n' "$IMAGE" "$PASS" "$FAIL"
if [[ $FAIL -gt 0 ]]; then
  printf 'failed:\n'; printf '  - %s\n' "${FAILED[@]}"
  exit 1
fi
echo "all tests passed"

[Copilot]

🟡 Changes recommended

The release workflow can republish an existing version tag on subsequent pushes to main, potentially mutating already-released artifacts (especially with SECURITY_REFRESH forcing rebuild differences).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates the cloudos-cli container build and release pipeline to reduce/avoid CRITICAL/HIGH vulnerabilities (Harbor) and to add a CI scan gate so releases only publish after passing the vulnerability check.

Changes:

• Rebased the Docker image on condaforge/miniforge3, added OS package upgrades, and removed specific scan-reported inventories/components.
• Updated the release workflow to build, scan with Trivy (CRITICAL/HIGH gate), then push artifacts; tag + GitHub Release are created at the end.
• Bumped version to 2.96.0 and added a changelog entry.

File summaries

File	Description
Dockerfile	Switches base image, updates OS packages, builds conda env, installs cloudos-cli, and applies scan-focused cleanups.
.github/workflows/release.yml	Adds a Trivy scan gate and reorders tagging/release creation to happen only after artifacts publish.
cloudos_cli/_version.py	Version bump to 2.96.0.
CHANGELOG.md	Adds release notes for v2.96.0.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Unresolved critical container-scanning and release-tagging issues remain.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

Dockerfile:57

• Passing a new SECURITY_REFRESH value does not invalidate this layer because the ARG is never referenced by the following RUN; Docker only includes an ARG in the cache key when it is used by an instruction. A cached final upgrade can therefore leave stale OS packages despite the workflow comment. Expand the argument in this RUN (or build with --no-cache) so the timestamp actually busts the cache.

.github/workflows/release.yml:132

• This pushes the version tag before the PyPI upload and before the Git tag marks the release complete. If a later step fails, the image remains publicly available under v$VER; a retry rebuilds with a new SECURITY_REFRESH value and can overwrite that same Quay tag (and :latest) with a different image, so the published container is not immutable. Claim/publish the version atomically or retain and republish the exact built digest on retries.

      - name: Push image to Quay (version tag)
        run: |
          set -euo pipefail
          VER="${{ needs.check.outputs.version }}"
          docker push "${QUAY_REPO}:v${VER}"

.github/workflows/release.yml:55

• The gate treats any existing tag as a completed release, but the tag is created before softprops/action-gh-release runs. If the release action fails after the tag push, a retry or later push sees RELEASE=false and permanently skips this version, leaving the GitHub Release absent even though the image and package may already be published. Make the gate verify/reconcile the GitHub Release, or make the final release creation retryable without using tag existence alone as the completion marker.

          if git rev-parse -q --verify "refs/tags/$TAG" >/dev/null; then
            echo "$TAG already exists: this version has shipped, nothing to do."
            echo "Bump __version__ in cloudos_cli/_version.py to cut a new release."
            echo "RELEASE=false" >> "$GITHUB_OUTPUT"

• Files reviewed: 6/6 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

A critical Dockerfile cleanup issue and moderate release workflow race conditions remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

Dockerfile:70

• SECURITY_REFRESH is never expanded in the following RUN, so changing --build-arg SECURITY_REFRESH=$(date +%s) does not invalidate this layer. Docker can reuse the cached apt-get upgrade, leaving the release scan to inspect an image without the latest security updates; reference the argument in the command (for example, with an echo before apt-get update).

.github/workflows/release.yml:152

• git ls-remote is only a preflight read; it does not reserve refs/tags/$TAG. Two runs outside this workflow's concurrency group can both pass it and then push the mutable v$VER image tag, allowing the later run to overwrite the image that was scanned. Since the surrounding comment explicitly calls out old/manual runs, use an atomic remote reservation or lock instead of relying on this check.

          if git ls-remote --exit-code --tags origin "refs/tags/$TAG" >/dev/null 2>&1; then
            echo "::error::$TAG appeared on origin while this job was building."
            echo "Another run has already released this version; refusing to"
            echo "overwrite ${QUAY_REPO}:$TAG with a different image."
            exit 1

.github/workflows/release.yml:180

• NEWEST is read from the tags fetched at the start of the release job, but the workflow explicitly acknowledges runs that may not be covered by its concurrency group. If a newer release tag is pushed after checkout, this stale list can make an older version pass the comparison and overwrite :latest, reversing release order. Refresh tags from origin immediately before this decision and add a final/serialized guard for publishing :latest; the check at lines 148-153 only protects the version tag.

          NEWEST="$(git tag --list 'v*' | sed 's/^v//' | sort -V | tail -1)"
          if [ -n "$NEWEST" ] \
             && [ "$(printf '%s\n%s\n' "$NEWEST" "$VER" | sort -V | tail -1)" != "$VER" ]; then
            echo "v$VER is older than the newest tag (v$NEWEST); leaving :latest unchanged."

• Files reviewed: 6/6 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🔵 Needs a closer look

Four moderate unresolved review findings remain in the Dockerfile and release workflow.

Review details

Suppressed comments (4)

Previously missed (1) — in code that hasn't changed since the last review.

Dockerfile:57

• SECURITY_REFRESH is never referenced by the following RUN; Docker does not include an unused ARG value in that instruction's cache key. Therefore the release workflow's --build-arg SECURITY_REFRESH=$(date +%s) will not invalidate a cached apt-get upgrade layer, so a reused build can ship stale OS packages despite the stated refresh guarantee. Reference the arg in the RUN (for example, with an echo) before the update.

.github/workflows/release.yml:180

• NEWEST includes pre-release tags even though the preceding branch deliberately excludes the current pre-release from :latest. If v2.97.0a1 exists and a stable patch v2.96.1 is then released, sort -V treats the pre-release as newer and this exits without moving :latest, leaving it on the older stable image. Filter NEWEST to final tags before comparing.

          NEWEST="$(git tag --list 'v*' | sed 's/^v//' | sort -V | tail -1)"
          if [ -n "$NEWEST" ] \
             && [ "$(printf '%s\n%s\n' "$NEWEST" "$VER" | sort -V | tail -1)" != "$VER" ]; then
            echo "v$VER is older than the newest tag (v$NEWEST); leaving :latest unchanged."

.github/workflows/release.yml:160

• The version image is published before the PyPI upload and final tag. If a later step fails, this leaves a public v${VER} image without a release, and a retry rebuilds with a new security-refresh value and can overwrite that same version tag with a different digest. Publish artifacts only after the preceding release steps succeed, or make retries reuse an immutable image digest.

      - name: Push image to Quay (version tag)
        run: |
          set -euo pipefail
          VER="${{ needs.check.outputs.version }}"
          docker push "${QUAY_REPO}:v${VER}"

.github/workflows/release.yml:218

• The Git tag is pushed before softprops/action-gh-release runs. If release creation fails, a rerun reaches the check job, sees the tag, and skips the entire release job, so the GitHub Release is never retried even though the image and PyPI artifacts may exist. Make the gate verify the GitHub Release too, or make finalization idempotent when the tag already exists.

            git tag -a "$TAG" -m "Release $TAG"
            git push origin "$TAG"
          fi

      - name: Create GitHub Release with auto-generated notes

• Files reviewed: 6/6 changed files
• Comments generated: 0 new
• Review effort level: Lite

[dapineyro]

LGTM!