diff --git a/.claude/skills/check-component/SKILL.md b/.claude/skills/check-component/SKILL.md new file mode 100644 index 000000000..c25488aae --- /dev/null +++ b/.claude/skills/check-component/SKILL.md @@ -0,0 +1,83 @@ +--- +name: check-component +description: >- + Verify a Viash component's change is fully deployed before (re)running a benchmark: + committed to origin/main, regenerated in build/main, and rebuilt as a fresh container + on ghcr (build_main tag). Use when a component edit doesn't seem to take effect on the + cluster / in a Nextflow or Seqera run, when a run errors with symptoms of an old image + (missing deps, wrong base image), or to confirm an image is not stale before launching. +--- + +# check-component + +A component change only reaches a benchmark run after **three** independent things +happen. A failure at any stage means the cluster keeps using the **old container**, so +the fix silently has no effect: + +1. **Source** — the change is committed and on `origin/main`. +2. **Codegen** — CI's `Build` workflow regenerates the `build/main` deploy branch + (the inlined `target/nextflow/.../main.nf` that run scripts launch via + `--revision build/main`). +3. **Container** — CI builds the component's Docker image and **pushes it to `ghcr.io`** + at the `build_main` tag. This is the step that most often lags or fails (heavy images, + source compiles, runner disk/time limits). Codegen redeploying does **not** imply the + container was rebuilt — they are separate steps. + +The decisive signal is the image's `org.opencontainers.image.revision` label: it records +the git commit the container was built from. Comparing it to `origin/main` HEAD tells you +whether the pushed image is current — read straight from the registry, without pulling the +(multi-GB) image. + +## How to run + +```bash +.claude/skills/check-component/check_component.sh +``` + +Accepts a bare component name (`segger`), a `namespace/component` path +(`methods_transcript_assignment/segger`), or a path to `config.vsh.yaml`. Optional 2nd arg +overrides the image tag (default `build_main`; for a `build/` deploy the tag is +`build_`). + +The script prints the three checks and a `VERDICT`. Report the verdict to the user and, when +something is stale, which stage failed and what to do about it. + +## Interpreting results + +- **All three OK** → a run with `--pull-latest` will use the current build. Safe to launch. +- **[1] source not on main** → the fix is uncommitted or on another branch (e.g. `fixes`). + Commit/merge to `main` first; the push to `main` is what triggers CI's `Build`. +- **[2] build/main behind** → the `Build` workflow hasn't redeployed the latest `main` yet. + It usually runs within ~1 min of the merge; if not, check the Actions "Build" run. +- **[3] container stale** (revision label ≠ `origin/main` HEAD) → the container push for the + latest commit hasn't succeeded. Either the `Build` run is still compiling the image, or the + image build **failed** (common for heavy GPU/compile images). Check the `Build` run for this + component; if it failed, rebuild + push the image manually: + + ```bash + # from a linux/amd64 host with docker + `docker login ghcr.io` + viash ns build -q --setup push + ``` + + (macOS/Apple-Silicon can't practically build amd64 CUDA images — use a Linux build host or + a build pod on the cluster.) + +- **[3] container `created-vs-commit: STALE`** (revision label **matches** HEAD, but the image + was **built before** the commit it is labelled with) → a **false-OK trap** the revision label + alone misses. The `revision` label was re-stamped onto old layers without a real rebuild — + Docker layer-cache reuse, a re-pushed/re-tagged manifest, or a heavy build that failed and + left the previous image in place. The deployed image does **not** contain the change even + though it claims the right SHA. Fix is the same as a plain stale container, but you must + **bust the cache** so the changed layers actually re-execute (build on a fresh builder, e.g. a + clean Docker-in-Docker pod, rather than one with warm cache). Tell: `image built` timestamp is + earlier than the revision commit's time. + +## Notes + +- Requires network access to the registry and `git fetch` access to `origin`. The script does + a `git fetch origin` itself. +- The container check queries `ghcr.io` with an anonymous pull token, which works for public + packages. For a private package or a non-ghcr registry the container stage reports + `NOT FOUND`; verify via `docker manifest inspect` with credentials instead. +- Project identity (organization, name, registry) is read from `_viash.yaml`; the image repo + path is `////`. diff --git a/.claude/skills/check-component/check_component.sh b/.claude/skills/check-component/check_component.sh new file mode 100644 index 000000000..08a2171fb --- /dev/null +++ b/.claude/skills/check-component/check_component.sh @@ -0,0 +1,169 @@ +#!/usr/bin/env bash +# check_component.sh — verify a Viash component's change is fully deployed. +# +# Runs three checks and prints a verdict: +# [1/3] source: is the component's config committed to origin/main? +# [2/3] codegen: did build/main redeploy the latest origin/main commit? +# [3/3] container: was the ghcr :build_main image rebuilt from the latest commit? +# +# The container check reads the image's org.opencontainers.image.revision label +# straight from ghcr (anonymous pull token) and compares it to origin/main HEAD, +# so it does NOT pull the (multi-GB) image. +# +# Usage: +# check_component.sh # e.g. segger +# check_component.sh / # e.g. methods_transcript_assignment/segger +# check_component.sh +# check_component.sh # tag defaults to build_main +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)" || { echo "ERROR: not in a git repo"; exit 2; } +cd "$REPO_ROOT" || exit 2 + +ARG="${1:-}" +TAG="${2:-build_main}" +[ -z "$ARG" ] && { echo "usage: check_component.sh [image-tag]"; exit 2; } + +# ---- resolve the component config path ---- +if [[ "$ARG" == *config.vsh.yaml && -f "$ARG" ]]; then + CFG="$ARG" +elif [[ -f "src/$ARG/config.vsh.yaml" ]]; then + CFG="src/$ARG/config.vsh.yaml" +elif [[ -d "$ARG" && -f "$ARG/config.vsh.yaml" ]]; then + CFG="$ARG/config.vsh.yaml" +else + HITS="$(find src -type d -name "$ARG" 2>/dev/null | while read -r d; do [ -f "$d/config.vsh.yaml" ] && echo "$d"; done)" + N="$(printf '%s\n' "$HITS" | grep -c . || true)" + if [ "$N" -eq 0 ]; then echo "ERROR: no component config found for '$ARG'"; exit 2; fi + if [ "$N" -gt 1 ]; then echo "ERROR: '$ARG' is ambiguous, matches:"; printf '%s\n' "$HITS"; exit 2; fi + CFG="$HITS/config.vsh.yaml" +fi + +COMP_DIR="$(dirname "$CFG")" # src// +COMP="$(basename "$COMP_DIR")" # +NS="${COMP_DIR#src/}"; NS="${NS%/$COMP}" # (may be nested) + +# ---- project identity from _viash.yaml ---- +ident() { grep -E "$1" _viash.yaml | head -1 | sed -E 's/.*:[[:space:]]*//; s/[[:space:]]*$//'; } +ORG="$(ident '^organization:')" +PROJ="$(ident '^name:')" +REG="$(ident 'docker_registry:')"; REG="${REG:-ghcr.io}" +IMG="$ORG/$PROJ/$NS/$COMP" # repo path after the registry + +echo "Component: $COMP ($COMP_DIR)" +echo "Image: $REG/$IMG:$TAG" +echo + +git fetch origin --quiet 2>/dev/null || echo "(warning: git fetch failed — results may be stale)" +MAIN_SHA="$(git rev-parse origin/main 2>/dev/null || echo '')" +MAIN_SUBJ="$(git log -1 --format=%s origin/main 2>/dev/null || echo '')" + +ok_src=1 ok_deploy=1 ok_cont=1 + +# ================= [1/3] source committed to main ================= +echo "[1/3] Source committed to origin/main" +echo " origin/main HEAD: ${MAIN_SHA:0:8} $MAIN_SUBJ" +UNCOMMITTED="$(git status --porcelain -- "$CFG" 2>/dev/null)" +if [ -n "$UNCOMMITTED" ]; then + echo " local uncommitted change: YES (working-tree edits to config not committed)" + ok_src=0 +else + echo " local uncommitted change: no" +fi +if git diff --quiet origin/main -- "$CFG" 2>/dev/null; then + echo " config matches origin/main: yes" +else + echo " config matches origin/main: NO (your version differs from what's on main)" + ok_src=0 +fi +[ "$ok_src" -eq 1 ] && echo " => OK: this component's config on origin/main matches your working tree" \ + || echo " => NOT on main: merge/commit the change before it can build" +echo + +# ================= [2/3] build/main regenerated ================= +echo "[2/3] build/main regenerated (codegen)" +DEPLOY_MSG="$(git log -1 --format=%s origin/build/main 2>/dev/null || echo '')" +DEPLOY_SHA="$(printf '%s' "$DEPLOY_MSG" | grep -oE '[0-9a-f]{40}' | head -1)" +echo " build/main last deploy: ${DEPLOY_SHA:0:8} ($DEPLOY_MSG)" +if [ -n "$DEPLOY_SHA" ] && [ "$DEPLOY_SHA" = "$MAIN_SHA" ]; then + echo " matches origin/main HEAD: yes" + echo " => OK: build/main codegen is current" +else + echo " matches origin/main HEAD: NO" + echo " => build/main has not redeployed the latest main yet (CI Build may be running)" + ok_deploy=0 +fi +echo + +# ================= [3/3] container rebuilt on ghcr ================= +echo "[3/3] Container rebuilt on $REG ($TAG)" +CONT_OUT="$(python3 "$SCRIPT_DIR/ghcr_inspect.py" "$REG" "$IMG" "$TAG" "$MAIN_SHA")" +C_STATUS="$(printf '%s\n' "$CONT_OUT" | awk '/^STATUS/{print $2}')" +if [ "$C_STATUS" != "ok" ]; then + C_DETAIL="$(printf '%s\n' "$CONT_OUT" | awk '/^DETAIL/{print $2}')" + echo " image on registry: NOT FOUND ($C_DETAIL)" + echo " => no $TAG image has been pushed for this component" + ok_cont=0 +else + C_REV="$(printf '%s\n' "$CONT_OUT" | awk '/^REVISION/{print $2}')" + C_CREATED="$(printf '%s\n' "$CONT_OUT" | awk '/^CREATED /{print $2}')" + C_CREATED_EPOCH="$(printf '%s\n' "$CONT_OUT" | awk '/^CREATED_EPOCH/{print $2}')" + C_BASE="$(printf '%s\n' "$CONT_OUT" | awk '/^BASE_CREATED/{print $2}')" + C_LAYERS="$(printf '%s\n' "$CONT_OUT" | awk '/^LAYERS/{print $2}')" + C_MATCH="$(printf '%s\n' "$CONT_OUT" | awk '/^MATCH/{print $2}')" + echo " image revision label: ${C_REV:0:8}" + echo " matches origin/main HEAD: $([ "$C_MATCH" = yes ] && echo yes || echo 'NO <-- STALE')" + echo " image built: $C_CREATED" + echo " base layer created: $C_BASE" + echo " layer count: $C_LAYERS" + if [ "$C_MATCH" = yes ]; then + echo " => OK: container was built from the current origin/main commit" + else + echo " => STALE: container was built from ${C_REV:0:8}, not ${MAIN_SHA:0:8}." + echo " The container push for the latest commit has not succeeded" + echo " (CI Build still running, or the image build failed)." + ok_cont=0 + fi + + # Cross-check: an honestly-built image is CREATED at (or after) the commit it + # claims to be built from. If 'created' predates the revision's commit time, the + # layers were NOT rebuilt for that revision — the label was re-stamped over old + # layers (Docker cache reuse, a re-pushed tag, or a heavy build that failed and + # left the previous image). The revision label alone can't catch this; the + # timestamp does. GRACE absorbs build-vs-git clock skew. + GRACE=120 + REV_EPOCH="$(git show -s --format=%ct "$C_REV" 2>/dev/null)" + # Show BOTH times in UTC. The image 'created' is UTC (…Z) but git's default + # commit time carries the author's zone — comparing them as-is is a classic + # mistake (an image looks "older" than its commit only because of the offset). + REV_UTC="$(TZ=UTC git show -s --date=format-local:'%Y-%m-%dT%H:%M:%SZ' --format=%cd "$C_REV" 2>/dev/null)" + if [ -n "$REV_EPOCH" ] && printf '%s' "$C_CREATED_EPOCH" | grep -qE '^[0-9]+$'; then + echo " revision commit time (UTC): $REV_UTC" + if [ "$C_CREATED_EPOCH" -lt "$((REV_EPOCH - GRACE))" ]; then + echo " created-vs-commit: STALE <-- image built BEFORE its own revision commit" + echo " => STALE: image 'created' ($C_CREATED, UTC) predates the commit ${C_REV:0:8}" + echo " it is labelled with ($REV_UTC). The revision label was stamped but the" + echo " layers were NOT rebuilt (cache reuse / re-stamped tag / failed heavy build)." + echo " Rebuild+push with a clean cache." + ok_cont=0 + else + echo " created-vs-commit: ok (image built after its revision commit)" + fi + else + echo " created-vs-commit: (skipped — revision commit not in local history)" + fi +fi +echo + +# ================= verdict ================= +if [ "$ok_src" -eq 1 ] && [ "$ok_deploy" -eq 1 ] && [ "$ok_cont" -eq 1 ]; then + echo "VERDICT: OK — fully deployed. A run with --pull-latest will use the current build." + exit 0 +else + echo "VERDICT: NOT fully deployed — a run now would use a stale build. See the failing stage above." + [ "$ok_src" -eq 0 ] && echo " - source not on main: commit/merge the change to main" + [ "$ok_deploy" -eq 0 ] && echo " - build/main behind: wait for / re-trigger the CI Build workflow" + [ "$ok_cont" -eq 0 ] && echo " - container stale: check the Build run for this component; if it failed, rebuild+push manually" + exit 1 +fi diff --git a/.claude/skills/check-component/ghcr_inspect.py b/.claude/skills/check-component/ghcr_inspect.py new file mode 100644 index 000000000..583f25188 --- /dev/null +++ b/.claude/skills/check-component/ghcr_inspect.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +"""Inspect a container image's build provenance from a registry without pulling it. + +Usage: ghcr_inspect.py + +Prints key/value lines consumed by check_component.sh: + STATUS ok|missing + DETAIL (only when missing) + REVISION org.opencontainers.image.revision label + CREATED + CREATED_EPOCH CREATED as unix seconds (empty if unparseable) + BASE_CREATED created time of the first (base) layer + LAYERS + MATCH yes|no REVISION == expected-git-sha +""" +import json +import re +import subprocess +import sys +from datetime import datetime, timezone + + +def to_epoch(iso): + """ISO8601 (with optional fractional seconds and trailing Z) -> unix seconds. + Fractional seconds are dropped; registry timestamps are UTC. '' if unparseable.""" + m = re.match(r"(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})", iso or "") + if not m: + return "" + y, mo, d, h, mi, s = (int(x) for x in m.groups()) + return str(int(datetime(y, mo, d, h, mi, s, tzinfo=timezone.utc).timestamp())) + +ACCEPT = ( + "application/vnd.oci.image.index.v1+json," + "application/vnd.oci.image.manifest.v1+json," + "application/vnd.docker.distribution.manifest.list.v2+json," + "application/vnd.docker.distribution.manifest.v2+json" +) + + +def curl(url, accept=None, token=None, follow=False): + cmd = ["curl", "-s"] + if follow: + cmd.append("-L") + if token: + cmd += ["-H", f"Authorization: Bearer {token}"] + if accept: + cmd += ["-H", f"Accept: {accept}"] + cmd.append(url) + return subprocess.run(cmd, capture_output=True, text=True).stdout + + +def jload(s): + try: + return json.loads(s) + except Exception: + return {} + + +def main(): + if len(sys.argv) < 5: + print("STATUS missing\nDETAIL bad-args") + return + reg, img, tag, main_sha = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4] + base = "https://ghcr.io" if reg.strip() == "ghcr.io" else f"https://{reg}" + + tok = jload(curl(f"{base}/token?scope=repository:{img}:pull&service={reg}")).get("token", "") + man = jload(curl(f"{base}/v2/{img}/manifests/{tag}", ACCEPT, tok)) + if not man or man.get("errors"): + code = man.get("errors", [{}])[0].get("code") if man else "NO_RESPONSE" + print(f"STATUS missing\nDETAIL {code}") + return + + if "manifests" in man: # multi-arch index -> prefer linux/amd64 + child = None + for m in man["manifests"]: + p = m.get("platform", {}) + if p.get("os") == "linux" and p.get("architecture") == "amd64": + child = m["digest"] + break + child = child or man["manifests"][0]["digest"] + man = jload(curl(f"{base}/v2/{img}/manifests/{child}", ACCEPT, tok)) + + cfg_digest = (man.get("config") or {}).get("digest", "") + if not cfg_digest: + print("STATUS missing\nDETAIL no-config-digest") + return + + cfg = jload(curl(f"{base}/v2/{img}/blobs/{cfg_digest}", token=tok, follow=True)) + labels = (cfg.get("config") or {}).get("Labels") or {} + rev = labels.get("org.opencontainers.image.revision", "") + if not rev: # fall back to scanning build history for the LABEL line + for h in cfg.get("history", []): + cb = h.get("created_by", "") + if "org.opencontainers.image.revision=" in cb: + rev = cb.split("org.opencontainers.image.revision=", 1)[1].split()[0].strip('"') + break + + hist = cfg.get("history", []) + base_created = hist[0].get("created", "") if hist else "" + match = "yes" if (rev and main_sha and rev == main_sha) else "no" + created = cfg.get("created", "") + print("STATUS ok") + print(f"REVISION {rev}") + print(f"CREATED {created}") + print(f"CREATED_EPOCH {to_epoch(created)}") + print(f"BASE_CREATED {base_created}") + print(f"LAYERS {len(man.get('layers', []))}") + print(f"MATCH {match}") + + +if __name__ == "__main__": + main() diff --git a/.claude/skills/debug-component-k8s/SKILL.md b/.claude/skills/debug-component-k8s/SKILL.md new file mode 100644 index 000000000..3752716e7 --- /dev/null +++ b/.claude/skills/debug-component-k8s/SKILL.md @@ -0,0 +1,153 @@ +--- +name: debug-component-k8s +description: >- + Troubleshoot a Viash component/module's RUNTIME environment by running its container + image live on the kubernetes cluster — dependency conflicts, missing/mismatched packages, + native-library (.so / UCX / cv2) errors, CLI/API changes in a pinned tool, base-image + problems. Use when a component fails INSIDE its image at run time (ImportError, + ModuleNotFoundError, version TypeError, undefined symbol, SIGABRT, a CLI "unknown option"), + and you need to reproduce and fix it without a full Nextflow/Seqera run. Complements + check-component (which only checks whether the image is deploy-fresh, not whether it works). +--- + +# debug-component-k8s + +`check-component` tells you *whether the right image is deployed*. This skill tells you +*whether that image actually works* — by running it on the cluster and poking at its +environment directly, so you find and validate a fix **before** paying for a ~30-min +rebuild and a GPU benchmark run. + +The core loop: **spin up a throwaway pod on the component's image → reproduce the failure +with small probes → iterate on a fix live (pip/conda/env) → encode the validated recipe in +`config.vsh.yaml` → rebuild.** Each real benchmark run and each rebuild is expensive, so +every fix should be proven in the pod first. + +## Prerequisites + +- `kubectl` pointed at the cluster (`kubectl config current-context`). On Nebius the token + expires — if a call prints "Switch to your browser to complete authentication", re-auth + (`nebius ...`) or the next long transfer/exec dies mid-stream. +- The component's image must exist on the registry (run `check-component ` first; if + it's stale/missing, fix that before debugging the wrong image). + +## How to run + +```bash +S=.claude/skills/debug-component-k8s/inspect_pod.sh +$S up # launch pod on :build_main, wait Ready (no GPU) +$S up --gpu # ...on a GPU node (only for rmm/cuInit or a real run) +$S up --scratch # ...with the shared scratch PVC mounted at /scratch +$S run '' # run a bash snippet inside the pod +$S py '' # run python (RAPIDS_NO_INITIALIZE preset so cudf imports) +$S down # delete the pod when done +``` + +Most dependency / import / native-lib / CLI troubleshooting needs **no GPU** — the default +CPU pod schedules anywhere and is fast. Only reach for `--gpu` to exercise code that touches +the device. + +## Inspecting a run's data files (`--scratch`) + +When you need to look at the actual `.h5ad` / `.zarr` a task staged or produced (shapes, +`.X`, `obs`/`var`, layers, why an assertion tripped), **run the component's OWN image** — +its stack (anndata, spatialdata, txsim, …) reads those files natively, so you never +pip-install readers into a generic pod. `--scratch` mounts the shared scratch PVC +(`tower-scratch`) at `/scratch`; because PVCs are namespace-scoped the pod then runs in the +PVC's namespace (`tower-nf`), and `run`/`py`/`down` find it there automatically. + +```bash +$S up moscot --scratch +$S py moscot 'import anndata as ad; a = ad.read_h5ad("/scratch//_viash_par/input_scrnaseq_reference_1/output_sc.h5ad"); print(a.X.shape, a.X.dtype, list(a.layers))' +$S down moscot +``` + +- **You must know the exact work-dir path — ASK THE USER for it.** Paths are opaque Nextflow + hashes (`/scratch///`); you cannot guess them. Given one, list it + first to orient (via the running pod, or a live `debug` pod that already mounts `/scratch`). +- Inside a task work dir, **inputs are staged under `_viash_par/_1/…`** (one dir per + API input; the file keeps its upstream name) and the **output** under the published + `mpii_.../cta_..output.h5ad` subpath. `.command.sh` lists the exact + `VIASH_PAR_*` paths, and `.command.err` / `.exitcode` show how the task ended. +- Pick the image whose deps match the file: an `.h5ad` → any component image (all have + anndata); a SpatialData `.zarr` → a component that installs `spatialdata`. +- For other clusters, override `SCRATCH_PVC` / `SCRATCH_NS` / `SCRATCH_MOUNT` in the env. +- Reading is fine even on non-root images — staged files are world-readable. + +## The diagnostic playbook + +Reproduce the actual error first, then work outward. Useful probes (adapt the package/module +names to the failure): + +- **Reproduce the import/run:** `$S py 'from X import Y'` — get the FULL traceback + (don't `2>/dev/null` it away). The deepest frame names the culprit module/symbol. +- **Versions + who-requires-what:** compare a package's installed version to what its + siblings *demand* — this is how you spot version conflicts: + ```python + import importlib.metadata as m + print(m.version("pandas")) + print(m.requires("anndata")) # -> ['pandas>=2.3', ...] (the conflicting pin) + ``` +- **Is it installed at all?** `python -c "import cugraph"` per module → `MISSING`/`OK`. +- **Read the pinned tool's source** (contracts drift on unpinned installs): grep the + installed package for the CLI/schema you depend on, e.g. + `grep -rn "segger_cell_id\|write_parquet" $(python -c "import segger,os;print(os.path.dirname(segger.__file__))")`. + Check a CLI's current options with ` --help`. +- **Native-library errors** (`libGL.so.1: cannot open`, `undefined symbol`, `double free`): + `ldd .so`, `nm -D .so | grep `, `find / -name 'libX.so*'` to spot a + **system lib shadowing a wheel's lib** (the classic cause — e.g. `/opt/hpcx/ucx` vs a pip + UCX wheel). Fix via the missing system pkg, a headless variant of the wheel, or + `LD_LIBRARY_PATH` ordering — but if it's a *heap* corruption (SIGABRT/double-free), a pin + won't save you; the base image is wrong (see below). +- **Validate a candidate fix live** before editing the config: `$S run 'pip install + "anndata>=0.12,<0.13" ... && python -c "..."'`. Re-check that the fix didn't disturb other + pins (`importlib.metadata.version` before/after). + +## Gotchas (learned the hard way) + +- **`sleep infinity`, not `sleep N`.** A finite sleep (or the image's own entrypoint) can let + the pod reach phase `Succeeded`; you then get `cannot exec into a completed pod`. The + helper's `up` already uses `sleep infinity`. +- **Long-lived `kubectl exec` streams time out** (`i/o timeout` / "reading from error stream") + over a flaky link. Keep execs SHORT. For long work (a build, a `docker save`, a poll), + detach it inside the pod with `setsid`/`nohup ... < /dev/null &` and write an exit-code + marker (`; echo $? > /work/done`), then poll the marker with short execs. `pgrep -f "cmd"` + false-matches your own poll command — check for a done-marker/file-size instead. +- **CPU-only pod ⇒ GPU init fails, and that's EXPECTED.** `numba: CUDA driver library cannot + be found`, `Failed to dlopen libcuda.so.1`, or a traceback ending in `rmm.reinitialize` / + `cuInit` just means *no GPU on this pod* — not a real bug. Distinguish it from genuine + errors; set `RAPIDS_NO_INITIALIZE=1 CUDF_NO_INITIALIZE=1 RMM_NO_INITIALIZE=1` (the `py` + helper does) so `import cudf` and friends don't abort. To test past it, use `--gpu`. +- **Non-root base images bite twice — at build AND at run.** Check `whoami`/`id -u`. + `rapidsai/base` runs as `rapids` (uid 1001). + - *Build:* a viash `type: apt` step FAILS (needs root). Install system-ish deps via conda + (`conda install -c conda-forge git`) and prefer wheels that avoid system libs (e.g. + `opencv-python-headless` instead of `opencv-python`, which drops the libGL dependency). + - *Run:* the container can't write Nextflow's **root-owned task scratch dir** → + `Permission denied` on `.command.log` / `.command.begin` / `.exitcode` (fails before your + script runs). viash can't set the image `USER`, so force root at the Nextflow level: add + `[runAsUser: 0]` to that process's `pod` directive (e.g. on the `gpu` label in + `labels_nebius.config`). Harmless for already-root images. +- **Don't infer from timestamps/labels — get ground truth.** When two signals disagree (a + label says current, the run says stale), *run the actual image* and read versions/imports. + Registry `created` timestamps are UTC; git commit times carry a zone — comparing them + raw is a classic false conclusion. +- **Pin unpinned upstream installs.** A `github: owner/repo` (no `@commit`) install tracks + trunk; the tool's CLI/output schema can change under your `script.py`. Pin to the commit you + validated (`owner/repo@`; get it from + `cat $(...)/segger-*.dist-info/direct_url.json`). +- **Heap corruption / SIGABRT from mixed native stacks ⇒ change the base image, not the pins.** + If a required library stack (e.g. full RAPIDS) fundamentally conflicts with the base's own + native libs (e.g. NGC's HPC-X UCX), stop patching — rebase on an image built for that stack + (e.g. `rapidsai/base`) and layer the rest on top. Validate the whole layered stack imports + (exit 0) in the pod before committing the new base. + +## After you find the fix + +1. Encode the validated recipe (pins, base image, extra installs) in the component's + `config.vsh.yaml` setup steps — **last-wins ordering matters** for pins that must beat a + github install's unpinned deps. +2. `viash config view src///config.vsh.yaml >/dev/null` to confirm it parses. +3. `$S down ` to clean up the pod. +4. Commit → merge to `main` → CI rebuilds → `check-component ` confirms the new image is + live → re-run. The pod only proves imports/coexistence; the GPU benchmark run is the final + confirmation for anything device-dependent. diff --git a/.claude/skills/debug-component-k8s/inspect_pod.sh b/.claude/skills/debug-component-k8s/inspect_pod.sh new file mode 100644 index 000000000..89de0d8e2 --- /dev/null +++ b/.claude/skills/debug-component-k8s/inspect_pod.sh @@ -0,0 +1,184 @@ +#!/usr/bin/env bash +# inspect_pod.sh — spin up a throwaway pod running a Viash component's container +# image on the CURRENT kubectl cluster, to troubleshoot its runtime environment +# (imports, dependency versions, native libraries, CLI) WITHOUT a full Nextflow / +# Seqera benchmark run. This is the interactive counterpart to `check-component` +# (which only checks deployment freshness). +# +# The image is resolved from _viash.yaml the same way the cluster pulls it: +# ////: +# +# Usage: +# inspect_pod.sh up [tag] [--gpu] [--scratch] # launch pod + wait Ready +# inspect_pod.sh run '' # run a bash snippet in the pod +# inspect_pod.sh py '' # run python (RAPIDS no-init env preset) +# inspect_pod.sh down # delete the pod +# inspect_pod.sh image [tag] # just print the resolved image ref +# +# tag defaults to build_main. Add --gpu to schedule on a GPU node (for tests that +# actually need a device — rmm/cuInit, a real run). Most dep/import/native-lib +# troubleshooting needs NO gpu, so omit it (faster, schedules anywhere). +# +# Add --scratch to mount the shared scratch PVC at $SCRATCH_MOUNT so you can inspect a +# Nextflow work dir's staged inputs/outputs (h5ad/zarr) using the COMPONENT'S OWN image — +# its real libraries (anndata/spatialdata/...) read those files natively, with no ad-hoc +# pip installs in a generic pod. PVCs are namespace-scoped, so the pod then runs in the +# PVC's namespace. You must know the exact work-dir path (an opaque hash) — ASK THE USER +# for it; it can't be guessed. Cluster specifics default to this repo's Seqera setup and +# can be overridden via env: +# SCRATCH_PVC (tower-scratch) SCRATCH_NS (tower-nf) SCRATCH_MOUNT (/scratch) +set -uo pipefail + +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)" || { echo "ERROR: not in a git repo"; exit 2; } +cd "$REPO_ROOT" || exit 2 + +CMD="${1:-}"; ARG="${2:-}" +[ -z "$CMD" ] || [ -z "$ARG" ] && { grep -E '^#( |$)' "$0" | sed -E 's/^# ?//'; exit 2; } + +# ---- resolve component dir -> namespace/component ---- +if [[ -d "src/$ARG" && -f "src/$ARG/config.vsh.yaml" ]]; then + COMP_DIR="src/$ARG" +else + HITS="$(find src -type d -name "$ARG" 2>/dev/null | while read -r d; do [ -f "$d/config.vsh.yaml" ] && echo "$d"; done)" + N="$(printf '%s\n' "$HITS" | grep -c . || true)" + [ "$N" -eq 0 ] && { echo "ERROR: no component '$ARG' under src/"; exit 2; } + [ "$N" -gt 1 ] && { echo "ERROR: '$ARG' is ambiguous:"; printf '%s\n' "$HITS"; exit 2; } + COMP_DIR="$HITS" +fi +COMP="$(basename "$COMP_DIR")" +NS="${COMP_DIR#src/}"; NS="${NS%/$COMP}" + +# ---- project identity from _viash.yaml ---- +ident() { grep -E "$1" _viash.yaml | head -1 | sed -E 's/.*:[[:space:]]*//; s/[[:space:]]*$//'; } +ORG="$(ident '^organization:')"; PROJ="$(ident '^name:')" +REG="$(ident 'docker_registry:')"; REG="${REG:-ghcr.io}" + +# ---- tag / flags ---- +TAG="build_main"; GPU=0; SCR=0 +for a in "${@:3}"; do + case "$a" in + --gpu) GPU=1 ;; + --scratch) SCR=1 ;; + -*) : ;; + *) TAG="$a" ;; + esac +done +IMG="$REG/$ORG/$PROJ/$NS/$COMP:$TAG" +POD="inspect-$(printf '%s' "$COMP" | tr '_' '-' | tr '[:upper:]' '[:lower:]')" + +# ---- scratch PVC config (namespace-scoped; override via env for other clusters) ---- +SCRATCH_PVC="${SCRATCH_PVC:-tower-scratch}" +SCRATCH_NS="${SCRATCH_NS:-tower-nf}" +SCRATCH_MOUNT="${SCRATCH_MOUNT:-/scratch}" + +# ---- GPU scheduling (override via env for other clusters) ---- +# GPU nodes on this Nebius cluster sit in a dedicated node group (see the `gpu` label in +# src/base/labels_nebius.config) and may carry a taint, so a bare `nvidia.com/gpu: 1` limit +# is not enough to place the pod — we also pin the nodeSelector and tolerate any taint. +GPU_NODESELECTOR_KEY="${GPU_NODESELECTOR_KEY:-nebius.com/node-group-id}" +GPU_NODESELECTOR_VAL="${GPU_NODESELECTOR_VAL:-mk8snodegroup-e00t775jb99svb7k5r}" + +# Resolve the pod's namespace by name: a plain pod lands in the current ns, but a +# --scratch pod lives in SCRATCH_NS. run/py/down use this so they hit it either way. +# (Guarded array expansion below keeps this working under bash 3.2 + `set -u`.) +pod_ns() { kubectl get pod -A --field-selector "metadata.name=$POD" -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null; } + +case "$CMD" in + image) echo "$IMG" ;; + + up) + echo "Component: $COMP ($NS) Pod: $POD" + echo "Image: $IMG" + UPNS=() + if [ "$SCR" -eq 1 ]; then + # PVCs are namespace-scoped, so the pod must live in the PVC's namespace. + echo "Scratch: PVC $SCRATCH_PVC -> $SCRATCH_MOUNT (namespace $SCRATCH_NS)" + UPNS=(-n "$SCRATCH_NS") + GPU_BLOCK="" + [ "$GPU" -eq 1 ] && GPU_BLOCK=" resources: + limits: + nvidia.com/gpu: 1" + # kubectl run can't attach a PVC volume; apply a manifest instead. Same + # `sleep infinity` rationale as below (don't let the pod complete). + cat <&1 | tail -1 +apiVersion: v1 +kind: Pod +metadata: + name: $POD + namespace: $SCRATCH_NS +spec: + restartPolicy: Never + containers: + - name: main + image: $IMG + command: ["sleep", "infinity"] +$GPU_BLOCK + volumeMounts: + - name: scratch + mountPath: $SCRATCH_MOUNT + volumes: + - name: scratch + persistentVolumeClaim: + claimName: $SCRATCH_PVC +YAML + elif [ "$GPU" -eq 1 ]; then + # Request a GPU + pin to the GPU node group and tolerate its taint. NOTE: `kubectl run + # --limits` was REMOVED in modern kubectl, so build the pod from a manifest (same + # `sleep infinity` rationale as below). Override GPU_NODESELECTOR_KEY/VAL for other + # clusters; clear them to rely on the resource limit alone. + NODESEL="" + [ -n "$GPU_NODESELECTOR_KEY" ] && NODESEL=" nodeSelector: + $GPU_NODESELECTOR_KEY: $GPU_NODESELECTOR_VAL" + cat <&1 | tail -1 +apiVersion: v1 +kind: Pod +metadata: + name: $POD +spec: + restartPolicy: Never +$NODESEL + tolerations: + - operator: "Exists" + containers: + - name: main + image: $IMG + command: ["sleep", "infinity"] + resources: + limits: + nvidia.com/gpu: 1 +YAML + else + # NOTE: use `sleep infinity` — a finite `sleep N` (or the image entrypoint) + # can let the pod complete (phase Succeeded), and you can't exec a completed pod. + kubectl run "$POD" --image="$IMG" --restart=Never \ + --command -- sleep infinity 2>&1 | tail -1 + fi + echo "waiting for Ready (image may be multi-GB; already-cached nodes are instant)..." + kubectl ${UPNS[@]+"${UPNS[@]}"} wait --for=condition=Ready "pod/$POD" --timeout=600s 2>&1 | tail -1 + kubectl ${UPNS[@]+"${UPNS[@]}"} get pod "$POD" -o wide 2>&1 | tail -1 + ;; + + run) + SNIP="${3:?need a bash snippet as the 3rd arg}" + PNS="$(pod_ns)"; NSARG=(); [ -n "$PNS" ] && NSARG=(-n "$PNS") + kubectl ${NSARG[@]+"${NSARG[@]}"} exec "$POD" -- bash -c "$SNIP" 2>&1 + ;; + + py) + SNIP="${3:?need a python snippet as the 3rd arg}" + PNS="$(pod_ns)"; NSARG=(); [ -n "$PNS" ] && NSARG=(-n "$PNS") + # preset RAPIDS no-init so `import cudf`/segger deps don't abort on a GPU-less pod + kubectl ${NSARG[@]+"${NSARG[@]}"} exec "$POD" -- bash -c ' + export RAPIDS_NO_INITIALIZE=1 CUDF_NO_INITIALIZE=1 RMM_NO_INITIALIZE=1 + python - <<'"'"'PYEOF'"'"' +'"$SNIP"' +PYEOF' 2>&1 + ;; + + down) + PNS="$(pod_ns)"; NSARG=(); [ -n "$PNS" ] && NSARG=(-n "$PNS") + kubectl ${NSARG[@]+"${NSARG[@]}"} delete pod "$POD" --wait=false 2>&1 + ;; + + *) echo "unknown command '$CMD' (up|run|py|down|image)"; exit 2 ;; +esac diff --git a/.claude/skills/document-method-troubleshooting/SKILL.md b/.claude/skills/document-method-troubleshooting/SKILL.md new file mode 100644 index 000000000..ce27441f8 --- /dev/null +++ b/.claude/skills/document-method-troubleshooting/SKILL.md @@ -0,0 +1,110 @@ +--- +name: document-method-troubleshooting +description: >- + Capture what was learned while troubleshooting a Viash method/metric component into a + durable two-layer record: an in-repo NOTES.md (the authoritative how-it-works / why-the- + setup-is-this-way / where-it-breaks reference, committed with the code) plus a memory file + (a short pointer + a dated iteration log). Use after debugging a component (build/deps/ + container/CLI/runtime issue), when the user says "document this", "write it up", "remember + the segger saga", "add to the notes", or asks to keep a running log of troubleshooting + iterations. Invoke again after EACH further iteration to append the log. +--- + +# document-method-troubleshooting + +Troubleshooting any component (segger, baysor, an atera loader, a metric, …) produces +hard-won knowledge — why the Docker base or a version pin is load-bearing, a subtle +logic/format/output-contract bug and its fix, what dead ends were ruled out, what still isn't +confirmed. That knowledge is worthless if it evaporates when the session ends or lives only +in a commit message nobody re-reads. This skill writes it down in **two layers that do not +duplicate each other**: + +| Layer | Where | Lives with | Contains | +|-------|-------|-----------|----------| +| **NOTES.md** | `src///NOTES.md` | the code, committed to git | the durable detail: how the component works step-by-step, why the setup/pins are the way they are, arguments, risk points/gotchas, wiring | +| **Memory** | auto-memory dir, `-.md` (type `project`) | the session (recalled automatically) | a short pointer to NOTES.md + the current one-paragraph state + a **dated iteration log** | + +**The rule that keeps them from drifting: deep detail goes in NOTES.md; the memory is a +pointer plus a running log.** If you're tempted to paste the Docker recipe rationale into the +memory, it belongs in NOTES.md. If you're tempted to put "tried X on 2026-07-23, still fails" +into NOTES.md, it belongs in the memory iteration log. + +## When invoked + +1. **Identify the component.** From the conversation or the user's words, resolve it to a + `src///` dir (e.g. `methods_transcript_assignment/segger`). Confirm + the dir exists. +2. **Check what already exists** — a `NOTES.md` in that dir, and a memory file for it + (search the auto-memory dir + `MEMORY.md` index). This decides create-vs-append below. +3. **Do the smaller of two jobs:** + - **First time / understanding changed** → write or revise `NOTES.md`, then create/refresh + the memory pointer. + - **Just another iteration** (tried something, got a result) → append one line to the + memory's iteration log; touch NOTES.md only if the *understanding* changed (a new root + cause, a new pin, a ruled-out dead end), not merely because an attempt was made. + +## NOTES.md — the in-repo reference + +Committed alongside the code so it survives context loss and helps the next person (human or +Claude). Write it as prose a maintainer would want before touching the component. **Scale it +to the component** — a method that needed one small fix gets a few honest paragraphs; a +fought-over one (many pins, dead ends, external CLI) gets the full treatment. Use the +sections that apply and drop the rest — don't pad a simple component to fit a big template. +Candidate sections: + +- **What this component is** — its stage/API (`src/api/comp_*` for methods; `comp_metric_*` + for metrics), what's unusual about it (GPU-only? R vs Python? wraps an external CLI? + adapter?), links (docs/repo/DOI). +- **script.py / script.R — step by step** — the real control flow, with `file:line` anchors, + and the *why* behind non-obvious choices. +- **Arguments** — table of args, defaults, and what they map to upstream. +- **Setup / Docker (often the highest-value section when it applies)** — why the base image, + why each version pin, any ORDER constraints, and the history: what was tried before and why + it failed (e.g. "NGC base → UCX collision → SIGABRT, unfixable by pip pins → rebased on + RAPIDS"). This is what stops someone re-breaking it. If the component just merges a base + setup and worked first try, say that in a line — no drama to record. +- **Wiring** — where it's registered (for a method: workflow config dep list + default + methods string, `main.nf` fan-out; metrics wire via the metrics workflow), the run/test + script. +- **Risk points / gotchas** — commit-pinned deps and why, output-contract assumptions, CI + coverage gaps, anything fragile. + +Keep `file:line` references honest — verify them against current code before writing them as +fact (they rot). State what is **validated** vs **not yet confirmed** explicitly. + +## Memory — the pointer + iteration log + +Follow the repo's memory conventions (see the memory instructions in the system context): +one file, frontmatter with `type: project`, body, and a one-line pointer added to `MEMORY.md`. +Structure the body as: + +- **One paragraph of current state** — what the component is, and the non-obvious facts + (load-bearing pins, commit pin, "not yet run end-to-end on GPU", CI gaps). Convert relative + dates to absolute. +- **A line that says NOTES.md is the authoritative detail** and this memory is the index/log, + so future-you reads NOTES.md for depth and doesn't try to keep both fully detailed. +- **`**How to apply:**`** — note that the log is appended after each iteration. +- **Links** — `[[other-memory]]` cross-refs to related components/lessons. +- **`## Iteration log`** — reverse-or-forward-chronological dated lines. **This is the part + that grows.** + +### Iteration log entry format + +One line per troubleshooting round: `- YYYY-MM-DD — → `. Be concrete and include dead ends (a ruled-out cause is as valuable as a fix): + +``` +- 2026-07-23 — memory created; state captured. Builds/imports validated on CPU (rmm mocked); + awaiting first confirmed GPU end-to-end run. +- 2026-07-24 — ran run_test_segger_nebius.sh on Nebius GPU → failed at with ; + suspect . Ruled out stale image (check-component: revision current). +``` + +When an iteration changes a durable fact (new pin, confirmed root cause, "now runs +end-to-end"), **also** update the state paragraph and, if it's about how the component works +or breaks, NOTES.md — then log that you did. + +## Output + +Tell the user, briefly: which files you wrote/appended, and (if this was an iteration) the +one-line log entry you added. Don't reprint the whole NOTES.md. diff --git a/.claude/skills/fetch-run-results/SKILL.md b/.claude/skills/fetch-run-results/SKILL.md new file mode 100644 index 000000000..7f9a63dc9 --- /dev/null +++ b/.claude/skills/fetch-run-results/SKILL.md @@ -0,0 +1,105 @@ +--- +name: fetch-run-results +description: >- + Given a Seqera/nebius workflow ID for a task_ist_preprocessing run, copy the run's + published result files (score_uns.yaml, method_configs.yaml, dataset_uns.yaml, state.yaml, + ... — the *.yaml the benchmark writes to publish_dir) off the cluster scratch PVC to a + local folder. Use when the user wants to fetch/download/extract/copy a run's result yamls + (or its whole output) to a certain folder, pull the scores for a finished run, or get the + published outputs of a sweep run locally by its workflow ID. Complements parse-sweep-results + (which reads the task list, not the output files). +--- + +# fetch-run-results + +Copy a finished run's **published output files** from the cluster to a local folder, keyed +off the **workflow ID** that `run_sweep` recorded in `/_runs.csv`. + +A benchmark run does not publish to S3 or anywhere the launch host can see — it writes its +results to `publish_dir` on the shared scratch PVC (`tower-scratch`, mounted at `/scratch`, +namespace `tower-nf`). So there are two steps, both automated here: + +1. **Ask Seqera** for the run's `publish_dir` (`tw runs view -i --params`). +2. **`kubectl cp`** the files out of a pod that mounts that PVC (the long-lived `debug` + pod by default; an ephemeral pod is spawned + torn down if `debug` is gone). + +This is the packaged version of the manual copy first done for the cellposev4 run +(`.../segmentation/cellpose4/`). + +## What lands in publish_dir + +A `run_benchmark` (`-entry auto`) run publishes these top-level yamls (plus a `trace.txt`): + +- **`score_uns.yaml`** — the metric **values** per `(dataset, method-combo)`. The one you + want to rank variants by. +- `method_configs.yaml` / `metric_configs.yaml` — the resolved configs that produced them. +- `dataset_uns.yaml`, `task_info.yaml`, `state.yaml` — dataset/run/output metadata. + +By default the skill copies all top-level `*.yaml`; `--all` grabs everything (incl. `trace.txt`). + +## How to run + +```bash +.claude/skills/fetch-run-results/fetch_results.sh [] \ + [--all] [--glob ''] [--workspace ] [--pod ] [--keep-pod] +``` + +- `` — the run id (e.g. `3JG1eHsWrAeSse`). +- `` — local folder to copy into (created if absent). The normal case is an + explicit folder, e.g. `~/projects/txsim_results/param_sweep_tests/segmentation/cellpose4`. + If omitted, defaults to `/runs/` (see `--base`). +- `--all` — copy the entire `publish_dir` (files + subdirs, structure preserved), not just + the top-level `*.yaml`. +- `--glob ''` — copy top-level files matching this glob instead of `*.yaml` + (e.g. `--glob 'score_uns.yaml'` for only the scores). +- `--workspace` — Seqera workspace id (default `$TOWER_WORKSPACE_ID` or `167877437119966`). +- `--pod` — pod that mounts the PVC to copy from (default `$SCRATCH_POD` or `debug`). +- `--keep-pod` — if an ephemeral pod was spawned, leave it up (default: delete it). + +Example (what the skill was built from): + +```bash +.claude/skills/fetch-run-results/fetch_results.sh 3JG1eHsWrAeSse \ + ~/projects/txsim_results/param_sweep_tests/segmentation/cellpose4 +``` + +## How it works + +1. **Resolve** `publish_dir` from the run's params (fails fast on auth / wrong id). +2. **Pick a pod** that mounts the scratch PVC: use `--pod`/`debug` if it's Running *and* can + `test -d publish_dir`; otherwise **spawn** an ephemeral pod (`fetch-results-`) that + mounts `tower-scratch` using the `process_dataset` image (resolved from `_viash.yaml`, so + it's a guaranteed-pullable ref), use it, and **tear it down** (unless `--keep-pod`). +3. **Copy** the matching files (or the whole dir with `--all`) into `` via + `kubectl cp`, then print the local listing. + +## Where results go (convention) + +Fits the same external results tree as the sibling skills (override with +`$TXSIM_RESULTS_DIR`), e.g. `~/projects/txsim_results/param_sweep_tests///`: + +- **run tables** → `/_runs.csv` ([`run_sweep`](../run_sweep/SKILL.md)) +- **parsed success/fail** → `/_run__results.csv` + ([`parse-sweep-results`](../parse-sweep-results/SKILL.md)) +- **fetched output yamls** (this skill) → whatever folder you point it at (e.g. + `//`), or `/runs/` by default. + +## Prerequisites & gotchas + +- **`tw` authenticated** to the workspace, and **`kubectl`** pointed at the cluster + (`kubectl config current-context` → `nebius-mk8s-...`). On Nebius the token expires; if a + call prints "Switch to your browser to complete authentication", re-auth first. +- **The `debug` pod is the fast path** — it's a long-lived pod in `tower-nf` that already + mounts `/scratch`, so copies are instant. The ephemeral-pod fallback pulls a multi-GB + image the first time on an uncached node (minutes), so keep `debug` around if you fetch often. +- **`--all` copies `trace.txt` too** (the Nextflow execution trace, useful for timings/OOM + diagnosis); the default `*.yaml` leaves it out. +- **"Fetched" ≠ "scored well."** `score_uns.yaml` holds the values; a combo present there + still may have scored poorly. To turn a run into a ran-vs-failed matrix instead, use + [`parse-sweep-results`](../parse-sweep-results/SKILL.md). + +## Output + +Report to the user: the resolved `publish_dir`, which files were copied (and from which pod, +noting if one was spawned/torn down), and the local destination path. Point out that +`score_uns.yaml` is where the metric values live if they want to rank variants next. diff --git a/.claude/skills/fetch-run-results/fetch_results.sh b/.claude/skills/fetch-run-results/fetch_results.sh new file mode 100644 index 000000000..a90384c03 --- /dev/null +++ b/.claude/skills/fetch-run-results/fetch_results.sh @@ -0,0 +1,158 @@ +#!/usr/bin/env bash +# fetch_results.sh — copy a Seqera/nebius run's published result files off the +# cluster scratch PVC to a local folder, keyed off the run's WORKFLOW ID. +# +# A benchmark run publishes its outputs to `publish_dir` on the shared scratch PVC +# (tower-scratch, mounted at /scratch), NOT to S3 and NOT anywhere the launch host +# can see. So to get them locally we: (1) ask Seqera for the run's publish_dir, and +# (2) `kubectl cp` the files out of a pod that mounts that PVC. +# +# Usage: +# fetch_results.sh [] [opts] +# +# the Seqera run id (e.g. 3JG1eHsWrAeSse), as recorded in _runs.csv. +# local folder to copy into (created if absent). Default: +# $TXSIM_RESULTS_DIR/runs/ (see --base). Giving an +# explicit folder is the normal case (e.g. .../segmentation/cellpose4). +# +# --all copy the ENTIRE publish_dir (all files + subdirs, structure preserved), +# not just top-level *.yaml (the default). Includes trace.txt etc. +# --glob copy top-level files matching this shell glob instead of '*.yaml' +# (e.g. --glob 'score_uns.yaml' for just the scores). Ignored with --all. +# --workspace Seqera workspace numeric id (default $TOWER_WORKSPACE_ID or 167877437119966). +# --pod pod that mounts the scratch PVC to copy from (default $SCRATCH_POD or 'debug'). +# --namespace namespace of that pod / PVC (default $SCRATCH_NS or 'tower-nf'). +# --pvc PVC to mount if we have to spawn our own pod (default $SCRATCH_PVC or 'tower-scratch'). +# --mount mount path for a spawned pod (default $SCRATCH_MOUNT or '/scratch'). +# --keep-pod if we spawned an ephemeral pod, leave it running (default: delete it). +# +# If the chosen pod is absent or can't see publish_dir, an ephemeral pod that mounts +# the PVC is spawned (using the process_dataset image, resolved from _viash.yaml), used, +# and torn down (unless --keep-pod). The long-lived `debug` pod is the fast path. +set -uo pipefail + +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)" || REPO_ROOT="$PWD" +cd "$REPO_ROOT" 2>/dev/null || true + +command -v tw >/dev/null 2>&1 || { echo "ERROR: 'tw' (Seqera CLI) not found on PATH"; exit 2; } +command -v kubectl >/dev/null 2>&1 || { echo "ERROR: 'kubectl' not found on PATH"; exit 2; } + +ID="" ; DEST="" ; MODE="yaml" ; GLOB="*.yaml" ; KEEP_POD=0 +WS="${TOWER_WORKSPACE_ID:-167877437119966}" +POD="${SCRATCH_POD:-debug}" +NS="${SCRATCH_NS:-tower-nf}" +PVC="${SCRATCH_PVC:-tower-scratch}" +MOUNT="${SCRATCH_MOUNT:-/scratch}" +BASE="${TXSIM_RESULTS_DIR:-$HOME/projects/txsim_results/param_sweep_tests}" + +while [ $# -gt 0 ]; do + case "$1" in + --all) MODE="all"; shift;; + --glob) GLOB="${2:?}"; shift 2;; + --workspace) WS="${2:?}"; shift 2;; + --pod) POD="${2:?}"; shift 2;; + --namespace) NS="${2:?}"; shift 2;; + --pvc) PVC="${2:?}"; shift 2;; + --mount) MOUNT="${2:?}"; shift 2;; + --base) BASE="${2:?}"; shift 2;; + --keep-pod) KEEP_POD=1; shift;; + -h|--help) sed -n '2,40p' "$0" | sed 's/^# \{0,1\}//'; exit 0;; + -*) echo "unknown option: $1"; exit 2;; + *) if [ -z "$ID" ]; then ID="$1"; elif [ -z "$DEST" ]; then DEST="$1"; + else echo "unexpected arg: $1"; exit 2; fi; shift;; + esac +done +[ -z "$ID" ] && { echo "usage: fetch_results.sh [] [opts]"; exit 2; } +export TOWER_WORKSPACE_ID="$WS" + +# ---- resolve publish_dir from Seqera ---- +echo "Resolving publish_dir for run $ID (workspace $WS) ..." +params="$(tw runs view -i "$ID" --params 2>&1)" \ + || { echo "$params"; echo "ERROR: could not fetch run $ID (auth? wrong id/workspace?)"; exit 1; } +PUBDIR="$(printf '%s\n' "$params" \ + | sed -nE 's/.*"?publish_dir"?[[:space:]]*[:=][[:space:]]*"?([^",]+)"?.*/\1/p' \ + | head -1 | sed -E 's/[[:space:]]+$//')" +[ -z "$PUBDIR" ] && { echo "ERROR: no publish_dir in the run's params. Params were:"; printf '%s\n' "$params" | head -40; exit 1; } +echo " publish_dir: $PUBDIR" + +# ---- default dest = /runs/ ---- +if [ -z "$DEST" ]; then + DEST="$BASE/runs/$(basename "$PUBDIR")" + echo " (no dest given) -> $DEST" +fi + +# ---- pick / spawn a pod that mounts the PVC and can see publish_dir ---- +SPAWNED=0 +pod_sees_dir() { kubectl exec -n "$NS" "$1" -- test -d "$PUBDIR" 2>/dev/null; } +pod_running() { [ "$(kubectl get pod "$1" -n "$NS" -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ]; } + +if pod_running "$POD" && pod_sees_dir "$POD"; then + echo "Using existing pod '$POD' (namespace $NS)." +else + if pod_running "$POD"; then + echo "Pod '$POD' is up but does not see $PUBDIR — spawning an ephemeral pod that mounts $PVC." + else + echo "Pod '$POD' not available — spawning an ephemeral pod that mounts $PVC." + fi + POD="fetch-results-${ID}" + # image ref from _viash.yaml (same resolution the cluster pulls with); guaranteed pullable. + ident() { grep -E "$1" _viash.yaml 2>/dev/null | head -1 | sed -E 's/.*:[[:space:]]*//; s/[[:space:]]*$//'; } + ORG="$(ident '^organization:')"; PROJ="$(ident '^name:')" + REG="$(ident 'docker_registry:')"; REG="${REG:-ghcr.io}" + IMAGE="${FETCH_IMAGE:-$REG/$ORG/$PROJ/data_processors/process_dataset:build_main}" + echo " image: $IMAGE" + cat <&1 | tail -1 +apiVersion: v1 +kind: Pod +metadata: + name: $POD + namespace: $NS +spec: + restartPolicy: Never + containers: + - name: main + image: $IMAGE + command: ["sleep", "infinity"] + volumeMounts: + - name: scratch + mountPath: $MOUNT + volumes: + - name: scratch + persistentVolumeClaim: + claimName: $PVC +YAML + SPAWNED=1 + echo " waiting for Ready (image may be multi-GB; cached nodes are instant)..." + kubectl wait --for=condition=Ready "pod/$POD" -n "$NS" --timeout=600s 2>&1 | tail -1 + pod_sees_dir "$POD" || { echo "ERROR: spawned pod still can't see $PUBDIR (wrong PVC/mount?)."; \ + [ "$KEEP_POD" -eq 0 ] && kubectl delete pod "$POD" -n "$NS" --wait=false >/dev/null 2>&1; exit 1; } +fi + +cleanup() { [ "$SPAWNED" -eq 1 ] && [ "$KEEP_POD" -eq 0 ] && \ + { echo "tearing down ephemeral pod '$POD' ..."; kubectl delete pod "$POD" -n "$NS" --wait=false >/dev/null 2>&1; }; } +trap cleanup EXIT + +mkdir -p "$DEST" + +# ---- copy ---- +if [ "$MODE" = all ]; then + echo "Copying ENTIRE publish_dir -> $DEST ..." + # kubectl cp : lands the dir's CONTENTS directly in (which we + # already mkdir'd), preserving any subdirs — not nested under /. + kubectl cp "$NS/$POD:$PUBDIR" "$DEST" 2>&1 | grep -v 'Removing leading' || true +else + files="$(kubectl exec -n "$NS" "$POD" -- bash -c "cd '$PUBDIR' 2>/dev/null && ls -1 $GLOB 2>/dev/null")" + [ -z "$files" ] && { echo "ERROR: no files matching '$GLOB' in $PUBDIR"; \ + echo " (dir listing:)"; kubectl exec -n "$NS" "$POD" -- bash -c "ls -1 '$PUBDIR'" 2>&1 | sed 's/^/ /'; exit 1; } + echo "Copying files matching '$GLOB' -> $DEST ..." + while IFS= read -r f; do + [ -z "$f" ] && continue + echo " $f" + kubectl cp "$NS/$POD:$PUBDIR/$f" "$DEST/$f" 2>&1 | grep -v 'Removing leading' || true + done <<< "$files" +fi + +# ---- report ---- +echo +echo "Done -> $DEST" +ls -la "$DEST" diff --git a/.claude/skills/make-dataset-config/SKILL.md b/.claude/skills/make-dataset-config/SKILL.md new file mode 100644 index 000000000..8b23039eb --- /dev/null +++ b/.claude/skills/make-dataset-config/SKILL.md @@ -0,0 +1,98 @@ +--- +name: make-dataset-config +description: >- + Create a Dataset_report dataset-paths YAML (like results/config/*.yaml) for a combined + dataset that lives on the cluster scratch PVC, and download its raw single-cell reference + into raw/ if that .h5ad is missing on the pod. Use when you need a per-dataset config for + the dataset visualisation report (the input to the render-qmd-pod skill), when asked to + "make the config/yaml for on the pod", or to stage a dataset's raw SC reference. + Writes results/config/.yaml with an id matching the dataset's id on the pod exactly, + auto-resolving the raw SC source from the combine scripts and pulling it from the public + openproblems-data S3 bucket when absent. +--- + +# make-dataset-config + +The dataset visualisation report needs three inputs — the **full** single-cell reference, +the **panel-subset** SC reference, and the **spatial** zarr. On the cluster the panel-subset +SC (`output_sc.h5ad`) and spatial (`output_sp.zarr`) already sit in the combined dataset dir, +but the **full raw SC reference** (the combine step's `input_sc`) is often **not staged** on +the PVC — only custom references (ltx/mpii/kuppe/ganier) are. This skill fills that gap: it +resolves the raw SC path from the combine scripts, **downloads it into `raw/` if missing**, +and writes a ready-to-render `results/config/.yaml`. Pair it with **render-qmd-pod** to +produce the HTML. + +Why the download is safe & 1:1: standard datasets set `input_sc` to +`$input_dir/` with `$input_dir=s3://openproblems-data/resources/datasets` — a public +bucket — so `` maps exactly onto the PVC (`raw/`) and onto the HTTPS endpoint +(`https://openproblems-data.s3.amazonaws.com/resources/datasets/`). + +## Prerequisites + +- `kubectl` on the cluster (`kubectl config current-context`). On Nebius the token expires — + re-auth if a call prints "Switch to your browser to complete authentication". +- A pod that **mounts the scratch PVC** at `/scratch` and has `curl` + `python3`/`h5py` + (default `debug` in `tower-nf`). Override with `SCRATCH_POD`; if none exists, bring one up + via the render-qmd-pod skill (`$S up`) and set `SCRATCH_POD=render-qmd`. + +## How to run + +```bash +S=.claude/skills/make-dataset-config/make_dataset_config.sh +$S # auto-resolve raw SC, download if missing, write config +$S --sc-source # override the S3 subpath under resources/datasets/ +$S --sc-raw # raw SC already on the PVC (custom ref); no download +$S --out # custom output path +``` + +`` is the dataset's dir under `datasets/` on the pod — the id in +`results/raw_dataset/datasets.csv` (e.g. `2024_10x_human_liver_xenium_combined`, or a rep like +`2023_10x_mouse_brain_xenium_combined/rep1`). Find ids with +`kubectl exec -n tower-nf debug -- ls /scratch/task_ist_preprocessing/datasets`. + +Example (the healthy-liver dataset — auto-resolves `2022Andrews_human_liver_sc`, downloads it +to `raw/` if absent, writes `results/config/2024_10x_human_liver_xenium_combined.yaml`): + +```bash +$S 2024_10x_human_liver_xenium_combined +``` + +Then render it: + +```bash +R=.claude/skills/render-qmd-pod/render_qmd.sh +$R up +$R render results/Dataset_report.qmd results/config/2024_10x_human_liver_xenium_combined.yaml \ + results/raw_dataset/2024_10x_human_liver_xenium_combined.html +$R down +``` + +## What it does + +1. Verifies `datasets//{output_sp.zarr,output_sc.h5ad}` exist on the pod. +2. Resolves the raw SC reference: parses `input_sc` for that id from + `scripts/create_resources/combine/*.sh`; `$input_dir/` → `raw/` + S3 URL. + (If it can't — e.g. a `$sc_ref` variable — it stops and asks for `--sc-source`/`--sc-raw`.) +3. Downloads the `.h5ad` into `raw/` **only if missing** (idempotent; atomic `.part`→final). +4. Best-effort checks the ref has `X_pca` + `normalized` + `cell_type` (what the report reads). +5. Writes `results/config/.yaml` (slashes in reps → `_`), with `id:` = the exact pod id, + `scrnaseq_raw` = the raw ref, and `dataset.dir` = the combined dir. + +## Gotchas + +- **The id must match the pod dir exactly.** Rep datasets carry a `/repN` (or `/B_…`, + `/978_reg1`) suffix — pass the full leaf id. The output filename slugifies `/`→`_`, but the + in-file `id:` and `dataset.dir` keep the true path. +- **Custom references (ltx/mpii/kuppe) aren't on S3** — their `input_sc` is a pod-local/`$sc_ref` + path, so auto-resolve can't form a URL. They're usually already on the PVC: pass + `--sc-raw raw//dataset.h5ad` (find it with + `kubectl exec -n tower-nf debug -- find /scratch/task_ist_preprocessing/raw -name dataset.h5ad`). +- **Big downloads land on shared storage.** Refs range ~380 MB–1.5 GB; the download is skipped + if the file is already there, so re-running is cheap. +- **`raw/` mirrors the S3 layout** (`raw/scrnaseq_for_ist//dataset.h5ad`) so a later + pipeline run or another config reuses the same file — don't invent a different raw path. + +## After you create the config + +- Render it with **render-qmd-pod** to confirm the ref + paths resolve end-to-end. +- The config is a repo file under `results/config/`; the downloaded `.h5ad` stays on the PVC. diff --git a/.claude/skills/make-dataset-config/make_dataset_config.sh b/.claude/skills/make-dataset-config/make_dataset_config.sh new file mode 100644 index 000000000..18db0df63 --- /dev/null +++ b/.claude/skills/make-dataset-config/make_dataset_config.sh @@ -0,0 +1,164 @@ +#!/usr/bin/env bash +# make_dataset_config.sh — create a Dataset_report dataset-paths YAML for a combined +# dataset that lives on the cluster scratch PVC, and DOWNLOAD its raw single-cell +# reference into raw/ if that .h5ad is missing on the pod. Writes results/config/.yaml +# with an id that matches the dataset's id on the pod exactly. +# +# The raw single-cell reference is the combine step's input_sc. For standard datasets +# that path is `$input_dir/` where `$input_dir=s3://openproblems-data/resources/ +# datasets`, so it mirrors 1:1 onto the PVC (`raw/`) and onto the public HTTPS +# endpoint (https://openproblems-data.s3.amazonaws.com/resources/datasets/). The +# helper resolves that mapping straight from the combine scripts; the spatial + panel-subset +# SC come from the combined outputs (output_sp.zarr / output_sc.h5ad) already on the PVC. +# +# Usage: +# make_dataset_config.sh [--sc-source ] [--sc-raw ] [--out ] +# +# the combined dataset's id = its dir under datasets/ on the pod +# (e.g. 2024_10x_human_liver_xenium_combined, or ".../rep1"). +# --sc-source override the raw-SC S3 subpath under resources/datasets/ (download-if-missing). +# Use when the combine-script auto-resolution can't (e.g. a $sc_ref var). +# --sc-raw the raw SC is ALREADY on the PVC at this raw-relative path (a custom +# reference, e.g. raw/ltx_human_lung_sc/.../dataset.h5ad); no download. +# --out output path (default results/config/_>.yaml). +# +# Env overrides (defaults target this repo's Nebius setup): +# SCRATCH_POD (debug) SCRATCH_NS (tower-nf) SCRATCH_MOUNT (/scratch) +# DATASET_ROOT ($SCRATCH_MOUNT/task_ist_preprocessing) CONFIG_DIR (results/config) +# S3_BASE (https://openproblems-data.s3.amazonaws.com/resources/datasets) +set -uo pipefail + +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)" || { echo "ERROR: not in a git repo"; exit 2; } +cd "$REPO_ROOT" || exit 2 + +ID=""; SC_SRC=""; SC_RAW=""; OUT="" +while [ $# -gt 0 ]; do + case "$1" in + --sc-source) SC_SRC="${2:?}"; shift 2;; + --sc-raw) SC_RAW="${2:?}"; shift 2;; + --out) OUT="${2:?}"; shift 2;; + -h|--help) grep -E '^#( |$)' "$0" | sed -E 's/^# ?//'; exit 0;; + -*) echo "unknown flag: $1"; exit 2;; + *) [ -z "$ID" ] && ID="$1"; shift;; + esac +done +[ -z "$ID" ] && { grep -E '^#( |$)' "$0" | sed -E 's/^# ?//'; exit 2; } + +POD="${SCRATCH_POD:-debug}" +NS="${SCRATCH_NS:-tower-nf}" +MOUNT="${SCRATCH_MOUNT:-/scratch}" +ROOT="${DATASET_ROOT:-$MOUNT/task_ist_preprocessing}" +S3_BASE="${S3_BASE:-https://openproblems-data.s3.amazonaws.com/resources/datasets}" +CONFIG_DIR="${CONFIG_DIR:-results/config}" + +xp() { kubectl exec -n "$NS" "$POD" -- sh -c "$1"; } + +kubectl get pod "$POD" -n "$NS" >/dev/null 2>&1 || { + echo "ERROR: pod '$POD' not found in namespace '$NS'. Point SCRATCH_POD at any pod that" + echo " mounts the scratch PVC (with curl + python3/h5py), or bring one up via the" + echo " render-qmd-pod skill (\$S up) and set SCRATCH_POD=render-qmd." + exit 2 +} + +echo "Dataset: $ID" +echo "Pod: $POD (ns $NS) root: $ROOT" + +# ── 1. the combined dataset dir must exist with its outputs ────────────────── +DDIR="datasets/$ID" +have="$(xp "cd '$ROOT/$DDIR' 2>/dev/null && ls -d output_sp.zarr output_sc.h5ad 2>/dev/null | tr '\n' ' '")" +case "$have" in + *output_sp.zarr*output_sc.h5ad*|*output_sc.h5ad*output_sp.zarr*) : ;; + *) echo "ERROR: $ROOT/$DDIR is missing output_sp.zarr and/or output_sc.h5ad (got: '${have:-none}')" + echo " Is the dataset id correct? List: kubectl exec -n $NS $POD -- ls $ROOT/datasets" + exit 3;; +esac +echo "Combined: $DDIR (output_sp.zarr + output_sc.h5ad present)" + +# ── 2. resolve the raw single-cell reference: RAW_REL (+ optional URL) ──────── +RAW_REL=""; URL="" +if [ -n "$SC_RAW" ]; then + RAW_REL="$SC_RAW" # already-on-PVC custom ref, no download +elif [ -n "$SC_SRC" ]; then + RAW_REL="raw/$SC_SRC"; URL="$S3_BASE/$SC_SRC" # explicit S3 subpath +else + # auto: pull input_sc for this id from the combine scripts (first match wins). + raw_sc_line="$(cat scripts/create_resources/combine/*.sh 2>/dev/null | awk -v id="$ID" ' + index($0, "id: \"" id "\"") {f=1} + f && /input_sc:/ {print; exit}')" + val="$(printf '%s' "$raw_sc_line" | sed -E 's/.*input_sc:[[:space:]]*"([^"]*)".*/\1/')" + case "$val" in + '$input_dir/'*|'${input_dir}/'*) + sub="${val#*input_dir\}/}"; sub="${sub#*input_dir/}" + RAW_REL="raw/$sub"; URL="$S3_BASE/$sub";; + *) + echo "ERROR: could not auto-resolve the raw SC reference for '$ID'." + echo " input_sc found: '${val:-}'" + echo " Pass --sc-source (to download from S3)," + echo " or --sc-raw ." + exit 3;; + esac +fi +echo "Raw SC: $RAW_REL" +[ -n "$URL" ] && echo " (S3: $URL)" + +# ── 3. download the raw SC .h5ad to raw/ if missing (idempotent) ───────────── +if xp "test -f '$ROOT/$RAW_REL'"; then + echo " already present ($(xp "du -h '$ROOT/$RAW_REL' 2>/dev/null | cut -f1"))" +elif [ -n "$URL" ]; then + echo " missing -> downloading from S3 ..." + xp "set -e; d=\$(dirname '$ROOT/$RAW_REL'); mkdir -p \"\$d\"; + curl -sSf '$URL' -o '$ROOT/$RAW_REL.part' && mv '$ROOT/$RAW_REL.part' '$ROOT/$RAW_REL'" \ + || { echo "ERROR: download failed from $URL"; exit 4; } + echo " downloaded ($(xp "du -h '$ROOT/$RAW_REL' 2>/dev/null | cut -f1"))" +else + echo "ERROR: raw SC ref missing at $ROOT/$RAW_REL and no download URL (custom --sc-raw)." + exit 4 +fi + +# ── 4. best-effort sanity check that the ref has what the report needs ─────── +kubectl exec -i -n "$NS" "$POD" -- python - "$ROOT/$RAW_REL" <<'PY' 2>/dev/null || echo " (skipped structure check: no python/h5py in $POD)" +import sys, h5py +p = sys.argv[1] +with h5py.File(p, "r") as f: + obsm = set(f["obsm"].keys()) if "obsm" in f else set() + layers = set(f["layers"].keys()) if "layers" in f else set() + obs = set(f["obs"].keys()) if "obs" in f else set() + miss = [n for n,ok in [("obsm/X_pca","X_pca" in obsm), + ("layers/normalized","normalized" in layers), + ("obs/cell_type", any(k=="cell_type" for k in obs))] if not ok] + print(" structure:", "OK (X_pca, normalized, cell_type)" if not miss + else "WARNING missing " + ", ".join(miss)) +PY + +# ── 5. write the config YAML ───────────────────────────────────────────────── +SLUG="$(printf '%s' "$ID" | tr '/' '_')" +OUT="${OUT:-$CONFIG_DIR/$SLUG.yaml}" +mkdir -p "$(dirname "$OUT")" +SRC_COMMENT="already staged on the PVC (custom reference)"; [ -n "$URL" ] && SRC_COMMENT="$URL" +cat > "$OUT" <- + Given a Seqera/nebius workflow ID for a task_ist_preprocessing parameter-sweep run, + parse its task list into a per-dataset x method-combo end-to-end success/failure table + (which param combos and which methods succeeded vs failed through to the final metric), + classify each failure (method-implementation error vs param-combo error) and FLAG for + troubleshooting only the methods that fail on their DEFAULT variant, optionally pulling the + real .command.err tracebacks. Saves under the results tree as /_run__*.csv. + Use when the user wants to parse/analyse a sweep run's results by its workflow ID, see which + combinations succeeded or failed, or triage/troubleshoot the failures. Complements run_sweep + (which launches the runs and records the IDs). +--- + +# parse-sweep-results + +Turn a finished (or running) sweep into a results table, keyed off the **workflow ID** that +`run_sweep` recorded in `/_runs.csv`. It pulls the run's tasks from Seqera +with `tw`, decides for every `(dataset, method-combo)` whether it **succeeded end-to-end** +(reached a **COMPLETED terminal metric** task `…/metric_`), and triages the failures. + +## How to run + +```bash +.claude/skills/parse-sweep-results/parse_run.sh \ + [--workspace ] [--stage ] [--method ] [--base ] [--keep-json] \ + [--diagnose [--all-failures] [--pod NAME] [--lines N]] +``` + +- `` — the run id (e.g. `3JG1eHsWrAeSse`). +- `--workspace` — Seqera workspace numeric id (default `$TOWER_WORKSPACE_ID` or `167877437119966`). +- `--stage` / `--method` — override the auto-inferred output folder / filename (normally + inferred from the swept token, e.g. `segm_cellposev4_*` → stage `segmentation`, method `cellposev4`). +- `--base` — results-tree base (default `$TXSIM_RESULTS_DIR` or `~/projects/txsim_results/param_sweep_tests`). +- `--diagnose` — for the **flagged** failures (methods that fail on their default variant), pull + the tail of each `.command.err` traceback off the scratch PVC via `kubectl exec` (default pod + `debug`, which has `tower-scratch` mounted). Saves to `_run__failures/*.err`. +- `--all-failures` — with `--diagnose`, fetch tracebacks for **every** failed combo (incl. param-only). +- `--pod NAME` / `--lines N` — pod to exec (default `debug`) / lines of traceback (default 40). + +## How it works + +Task **tags** encode the whole pipeline config `"///…/metric_"` +(control = `"/control_/metric_"`). The script: + +1. Fetches the run status headline, then **paginates all tasks** (`tw` caps at 100/page). +2. **Auto-detects the swept axis** = the chain-token position whose value varies most across + non-control tasks — works for any stage, for a single method's param sweep OR several methods + at one stage (each method → `param=default`). +3. **Parses each varying token** into `method / param / value` (base-method = the token-prefix + shared by the most variants; a different method used as the fixed baseline is its own method). +4. Marks each `(dataset, combo)` **SUCCEEDED** iff it has a COMPLETED `metric_*` task, else **FAILED**. +5. **Triages + classifies** every failure (see below), writes a `*_failures.csv`, and with + `--diagnose` fetches the tracebacks. + +### Truncation gotcha (important) + +Seqera **truncates long task tags**, so the trailing `/metric_` is chopped for long combos +(worst on long dataset names). Detect terminal-metric tasks by **process name** (discovered from +the short tags that still show a `metric_` segment), never by tag suffix. The swept token near the +front of the tag survives, so combo identity and success are still recovered correctly. + +## Classifying failures (triage) + +The point of the failure analysis is to separate **fix-the-code** from **bad-input-param**: + +- **🚩 TROUBLESHOOT = a method that fails on its DEFAULT variant.** This is the primary triage + rule: if the method breaks out-of-the-box on a dataset, that's a real implementation/robustness + problem worth chasing. Methods whose default passes and only a *non-default param* fails are + **not flagged** (the sweep is *supposed* to probe bad params) — they're recorded as param-only. + In a multi-method run (no params) every method's sole variant IS its default, so any failure flags it. + +Each failed combo also gets a heuristic `likely_class` to explain the mechanism: + +- **where it dies:** FAILED in the combo's **own method step** → suspect the method/impl; FAILED in + a **downstream** step (the method produced output, a later stage choked) → a param yielded + degenerate output (e.g. empty segmentation → downstream divide-by-zero). +- **failure ratio:** if ~all of a method's variants fail on a dataset → *method/dataset-level* (impl + bug or data-incompat, not one param); if only this variant fails → *param-specific*. + +**Deploy-state caveat:** an "impl error" verdict may just be an **undeployed fix** — the run uses the +`build/main` container, which can lag your working tree. Before blaming the code, cross-check with the +`check-component` skill (this is real: the stardist `min_overlap` fix existed locally but the run ran +the old container). `run_sweep`'s build gate exists precisely to avoid launching onto a stale build. + +**Retry semantics** (`src/base/labels_nebius.config`): exit-1 (code error) is retried once (2 attempts), +OOM (137/139) up to 3, then `ignore` so the run completes. A **FAILED** combo therefore failed on +**every** attempt — not transient. + +## Output (under `//`) + +- `_run__results.csv` — `dataset,combo,stage,method,param,value,reached_metric,end_to_end_status`. +- `_run__failures.csv` (only if there are failures) — + `dataset,combo,method,param,value,is_default,failing_step,exit,task_id,troubleshoot,likely_class` + (`troubleshoot=yes` ⇔ the method fails on its default variant). +- `_run__failures/*.err` (with `--diagnose`) — the `.command.err` tail per fetched failure. + +The console prints the run headline, a combos×datasets PASS/FAIL matrix, and a FAILURE DIAGNOSIS +listing the flagged (`[FLAG]`) methods and each failure's step/exit/`likely_class`. + +Report to the user: overall status, per-dataset pass/fail counts, **which methods are flagged for +troubleshooting (fail on default)** vs the param-only failures, and the CSV paths. + +## Notes & caveats + +- **"Succeeded" = ran to completion, not "scored well."** To rank passing combos by metric *value*, + use the `fetch-run-results` skill to pull `score_uns.yaml` (offer as a follow-up). +- Multi-axis (hand-built) runs fall back to labelling combos by the full varying-token tuple. +- Requires `tw` (authenticated), `jq`, `python3`; `--diagnose` also needs `kubectl` + a pod with + `tower-scratch` mounted. diff --git a/.claude/skills/parse-sweep-results/parse_run.py b/.claude/skills/parse-sweep-results/parse_run.py new file mode 100644 index 000000000..c161b278a --- /dev/null +++ b/.claude/skills/parse-sweep-results/parse_run.py @@ -0,0 +1,315 @@ +#!/usr/bin/env python3 +"""Analyse a Seqera run's task list into a per-dataset x pipeline-combo end-to-end +success/failure table for a task_ist_preprocessing benchmark run. + +Works for ANY run, not just single-method parameter sweeps: + * one method swept over parameters -> combos = param variants + * several methods at one stage -> combos = the methods (param "default") + * mixed / multiple varying stages -> combos = the tuple of varying tokens + +Reads the task JSON (list of {status, tag, process, exit}) fetched by parse_run.sh. +Task tags encode the pipeline config as + ///.../metric_ +controls appear as /control_/metric_. + +"succeeded in the end" == the combo has a COMPLETED terminal-metric task. + +Two Seqera facts this handles: + * task TAGS are TRUNCATED, so the trailing "/metric_" is chopped for long combos + -> metric tasks are detected by PROCESS name (never truncated), discovered from the + short tags that still show a metric_ segment. + * the swept/ varying token sits near the FRONT of the tag, which survives truncation. +""" +import argparse, json, os, sys +from collections import defaultdict + +OK = {"COMPLETED", "CACHED"} +# stage-token prefix -> results-tree folder +STAGE_FOLDER = { + "segm": "segmentation", "ass": "transcript_assignment", + "cta": "cell_type_annotation", "corr": "expression_correction", + "aggr": "count_aggregation", "qc": "qc_filter", "cell": "volume", + "norm": "normalization", "gene": "gene_efficiency", +} + + +def chain_of(tag): + return tag.split("/")[1:] + + +def combo_chain(tag): + """tokens after the dataset, with any metric_ segment removed.""" + return [s for s in chain_of(tag) if not s.startswith("metric_")] + + +def is_control(ch): + return any(s.startswith("control_") for s in ch) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--tasks", required=True) + ap.add_argument("--id", required=True) + ap.add_argument("--base", required=True) + ap.add_argument("--stage", default=None) + ap.add_argument("--method", default=None) + ap.add_argument("--overall", default="?") + ap.add_argument("--run-name", default="?") + args = ap.parse_args() + + tasks = json.load(open(args.tasks)) + if not tasks: + print("No tasks for run", args.id, "(still queued, or wrong id/workspace)."); sys.exit(1) + + # ---- metric tasks by PROCESS (truncation-proof) ---- + metric_procs = {t.get("process", "") for t in tasks + if any(s.startswith("metric_") for s in chain_of(t["tag"]))} + + def variance(drop_last): + """distinct-token sets per chain index over non-metric, non-control tasks.""" + idx = defaultdict(set) + for t in tasks: + if t.get("process", "") in metric_procs: + continue + ch = combo_chain(t["tag"]) + if is_control(ch): + continue + toks = ch[:-1] if (drop_last and len(ch) > 1) else ch + for i, tok in enumerate(toks): + idx[i].add(tok) + return idx + + # Drop each chain's LAST token first (it may be a truncated tail); if that finds no + # variation (e.g. everything failed at the first stage), retry without dropping. + idx_vals = variance(drop_last=True) + V = sorted(i for i, s in idx_vals.items() if len(s) > 1) + if not V: + idx_vals = variance(drop_last=False) + V = sorted(i for i, s in idx_vals.items() if len(s) > 1) + multi = len(V) > 1 + + def key_of(ch): + if not V: + return ("(all-default)",) + return tuple(ch[i] if i < len(ch) else "?" for i in V) + + # ---- aggregate per (dataset, combo) ---- + attempted = defaultdict(lambda: {"metric_ok": False, "metric_any": False}) + controls = defaultdict(lambda: {"metric_ok": False, "metric_any": False}) + failures = [] # raw non-OK: (ds, chain, step, status, exit, taskId) + failinfo = defaultdict(list) # (ds, combo-key) -> [(taskId, step, exit)] of its non-OK tasks + for t in tasks: + tag, st, proc = t["tag"], t.get("status", "?"), t.get("process", "") + tid = t.get("taskId") + ds = tag.split("/")[0] + ch = combo_chain(tag) + is_metric = proc in metric_procs + step = proc.split(":")[-1].replace("_process", "") # e.g. stardist, basic_qc_filter + if st not in OK: + failures.append((ds, "/".join(chain_of(tag)), step, st, t.get("exit"), tid)) + ctrl = next((s for s in ch if s.startswith("control_")), None) + if ctrl: + c = controls[(ds, ctrl)] + if is_metric: + c["metric_any"] = True; c["metric_ok"] |= st in OK + continue + k = key_of(ch) + if "?" in k: + continue # auxiliary task (e.g. extract_uns_metadata) with no combo token + a = attempted[(ds, k)] + if is_metric: + a["metric_any"] = True; a["metric_ok"] |= st in OK + if st not in OK: + failinfo[(ds, k)].append((tid, step, t.get("exit"))) + + datasets = sorted({d for (d, _) in attempted} | {d for (d, _) in controls}) + combos = sorted({k for (_, k) in attempted}) + + # ---- label combos + parse method/param/value (single varying axis) ---- + def stage_of(tok): + return STAGE_FOLDER.get(tok.split("_")[0], tok.split("_")[0]) + + single = (len(V) == 1) + parsed = {} # key-tuple -> (stage, method, param, value, label) + if single: + toks = [k[0] for k in combos] + prefix = toks[0].split("_")[0] if toks else "" + def rem(tok): + return tok[len(prefix) + 1:] if tok.startswith(prefix + "_") else tok + rems = [rem(t) for t in toks] + # a remainder is a "base method" if no OTHER remainder is a proper prefix of it + base = [r for r in rems if not any(o != r and r.startswith(o + "_") for o in rems)] + def parse(tok): + r = rem(tok) + cand = [m for m in base if r == m or r.startswith(m + "_")] + method = max(cand, key=len) if cand else r + if r == method: + return stage_of(tok), method, "default", "-" + rest = r[len(method) + 1:] + p, v = rest.rsplit("_", 1) if "_" in rest else (rest, "-") + return stage_of(tok), method, p, v + for k in combos: + s, m, p, v = parse(k[0]) + label = m if p == "default" else f"{m} {p}={v}" + parsed[k] = (s, m, p, v, label) + methods_with_params = sorted({parsed[k][1] for k in combos if parsed[k][2] != "default"}) + stage_folder = args.stage or (parsed[combos[0]][0] if combos else "misc") + fname_method = args.method or (methods_with_params[0] if len(methods_with_params) == 1 + else stage_folder) + else: + for k in combos: + label = " | ".join(k) + parsed[k] = ("misc", "-", "-", "-", label) + # infer stage folder from the first varying index's tokens + first_tok = combos[0][0] if combos and combos[0][0] not in ("(all-default)", "?") else "" + stage_folder = args.stage or (stage_of(first_tok) if first_tok else "misc") + fname_method = args.method or stage_folder + + # ---- write CSV ---- + out_dir = os.path.join(args.base, stage_folder) + os.makedirs(out_dir, exist_ok=True) + csv_path = os.path.join(out_dir, f"{fname_method}_run_{args.id}_results.csv") + rows = [] + for (ds, k) in sorted(attempted): + s, m, p, v, _ = parsed[k] + a = attempted[(ds, k)] + rows.append((ds, " | ".join(k), s, m, p, v, + "yes" if a["metric_any"] else "no", + "SUCCEEDED" if a["metric_ok"] else "FAILED")) + for (ds, ctrl) in sorted(controls): + c = controls[(ds, ctrl)] + rows.append((ds, ctrl, "control", ctrl[len("control_"):], "control", "-", + "yes" if c["metric_any"] else "no", + "SUCCEEDED" if c["metric_ok"] else "FAILED")) + with open(csv_path, "w") as f: + f.write("dataset,combo,stage,method,param,value,reached_metric,end_to_end_status\n") + for r in rows: + f.write(",".join(str(x) for x in r) + "\n") + + # ---- classify failures: method-implementation error vs param-combo error ---- + # Heuristics (see SKILL.md "Classifying failures"): + # * where it dies: FAILED in the combo's OWN method step -> suspect the method/impl; + # FAILED in a DOWNSTREAM step -> the method ran, a param produced degenerate output. + # * failure ratio: if ~all of a method's variants fail on a dataset, it's method/dataset + # level (not one param); if only this variant fails, it's param-specific. + ds_method_total = defaultdict(int); ds_method_fail = defaultdict(int) + for (ds, k) in attempted: + m = parsed[k][1] + ds_method_total[(ds, m)] += 1 + if not attempted[(ds, k)]["metric_ok"]: + ds_method_fail[(ds, m)] += 1 + + def classify(ds, k): + m = parsed[k][1] + info = failinfo.get((ds, k), []) + steps = sorted({s for (_, s, _) in info}) + tid = info[0][0] if info else None + ex = info[0][2] if info else "?" + own = m in steps # failed in its own method step + tot, fl = ds_method_total[(ds, m)], ds_method_fail[(ds, m)] + if not steps: + cls = "no-failing-task (metric missing; upstream/aux or truncation)" + elif not own: + cls = f"PARAM-COMBO -> degenerate output; crashes downstream in {'/'.join(steps)}" + elif tot > 1 and fl >= max(2, tot - 1): + cls = (f"METHOD/DATASET-level: {fl}/{tot} of {m}'s combos fail here " + f"-> impl bug or data-incompat (not one param)") + else: + cls = f"PARAM-SPECIFIC at method step ({'/'.join(steps)}) -> bad param or fragile handling" + return steps, tid, ex, cls.replace(",", ";") + + fail_combos = [(ds, k) for (ds, k) in sorted(attempted) if not attempted[(ds, k)]["metric_ok"]] + + # ---- TRIAGE: flag for troubleshooting ONLY methods that fail on their DEFAULT variant ---- + # A default-variant failure = the method is broken out-of-the-box on that data (a real + # implementation / robustness issue worth troubleshooting). A failure only on a NON-default + # param = the param choice / a fragile edge — recorded, but NOT flagged. (In a multi-method + # run every method's sole variant IS its default, so any such failure flags that method.) + method_default_fail = set() + for (ds, k) in attempted: + _, m, p, _, _ = parsed[k] + if p == "default" and not attempted[(ds, k)]["metric_ok"]: + method_default_fail.add(m) + methods_with_fail = {parsed[k][1] for (ds, k) in fail_combos} + flagged_methods = sorted(method_default_fail) + param_only_methods = sorted(methods_with_fail - method_default_fail) + + fail_csv = os.path.join(out_dir, f"{fname_method}_run_{args.id}_failures.csv") + ft_path = os.path.join(out_dir, f".failed_tasks_{args.id}.tsv") # hand-off for parse_run.sh --diagnose + if fail_combos: + with open(fail_csv, "w") as f, open(ft_path, "w") as ft: + f.write("dataset,combo,method,param,value,is_default,failing_step,exit,task_id,troubleshoot,likely_class\n") + for (ds, k) in fail_combos: + s, m, p, v, _ = parsed[k] + steps, tid, ex, cls = classify(ds, k) + flag_method = m in method_default_fail # method-level -> CSV `troubleshoot` + flag_diag = (p == "default") # this row is a default-variant failure + f.write(",".join(str(x) for x in + (ds, " | ".join(k), m, p, v, "yes" if p == "default" else "no", + "/".join(steps) or "?", ex, tid, "yes" if flag_method else "no", cls)) + "\n") + if tid is not None: # --diagnose default fetches only default-variant failures (flag_diag) + ft.write(f"{tid}\t{ds}\t{' | '.join(k)}\t{'/'.join(steps)}\t{ex}\t{1 if flag_diag else 0}\n") + + # ---- report ---- + n_ok = sum(1 for r in rows if r[7] == "SUCCEEDED") + print(f"Run {args.id} ({args.run_name}) overall={args.overall}") + axis = "none" if not V else (f"single (chain index {V[0]})" if single else f"MULTIPLE {V}") + print(f"datasets: {len(datasets)} | varying axis: {axis} | combos/dataset: {len(combos)}") + if single: + print(f"methods: {sorted({parsed[k][1] for k in combos})}") + print(f"tasks: {len(tasks)} | non-OK tasks: {len(failures)}") + print(f"rows (incl. controls): {len(rows)} -> SUCCEEDED {n_ok} FAILED {len(rows) - n_ok}") + print(f"CSV: {csv_path}\n") + + # matrix: combos (rows) x datasets (cols) + labels = {k: parsed[k][4] for k in combos} + ctrl_ks = sorted({c for (_, c) in controls}) + width = max([len(labels[k]) for k in combos] + [len("control: " + c[8:]) for c in ctrl_ks] + [12]) + ds_short = [d.replace("_test_combined", "") for d in datasets] + print(" " + "combo".ljust(width) + " " + " ".join(s[:22].center(22) for s in ds_short)) + def cell(present, ok): + return ("PASS" if ok else "FAIL") if present else " - " + for k in combos: + line = " " + labels[k].ljust(width) + for ds in datasets: + a = attempted.get((ds, k)) + line += " " + cell(a is not None, a and a["metric_ok"]).center(22) + print(line) + for c in ctrl_ks: + line = " " + ("control: " + c[8:]).ljust(width) + for ds in datasets: + cc = controls.get((ds, c)) + line += " " + cell(cc is not None, cc and cc["metric_ok"]).center(22) + print(line) + + if fail_combos: + print(f"\nFAILURE DIAGNOSIS ({len(fail_combos)} failed combos; {len(failures)} non-OK task records):") + if flagged_methods: + print(f" >> TROUBLESHOOT (fail on DEFAULT variant): {', '.join(flagged_methods)}") + if param_only_methods: + print(f" .. param-only failures (NOT flagged): {', '.join(param_only_methods)}") + print() + for (ds, k) in fail_combos: + _, m, _, _, lab = parsed[k] + steps, tid, ex, cls = classify(ds, k) + mark = "[FLAG]" if m in method_default_fail else " - " + print(f" {mark} {ds.replace('_test_combined','')} | {lab}") + print(f" step={'/'.join(steps) or '?'} exit={ex} taskId={tid} -> {cls}") + print(f"\n Failures CSV: {fail_csv}") + print(" Retry note: exit-1 combos get 2 attempts, OOM(137/139) up to 3, then 'ignore' —") + print(" so a FAILED combo failed on ALL attempts. Re-run with --diagnose for tracebacks") + print(" (flagged/fail-on-default only; add --all-failures to include param-only ones).") + print(" Tip: an impl-error verdict may just be an UNDEPLOYED fix — cross-check with") + print(" check-component before blaming the code.") + if multi: + print("\nNOTE: multiple varying axes detected — combos are labelled by the full " + "tuple of varying tokens; deep tokens may be affected by tag truncation.") + print(f"\nOUT_DIR={out_dir}") + if fail_combos: + print(f"FAILURES_CSV={fail_csv}") + print(f"FAILED_TASKS={ft_path}") + print(f"WROTE_CSV={csv_path}") + + +if __name__ == "__main__": + main() diff --git a/.claude/skills/parse-sweep-results/parse_run.sh b/.claude/skills/parse-sweep-results/parse_run.sh new file mode 100644 index 000000000..850d63da6 --- /dev/null +++ b/.claude/skills/parse-sweep-results/parse_run.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# parse_run.sh — parse a Seqera/nebius sweep run into a per-dataset x combo +# end-to-end success/failure table, saved under the results tree by pipeline stage. +# +# Usage: +# parse_run.sh [--workspace ] [--stage ] [--method ] +# [--base ] [--keep-json] [--diagnose [--pod NAME] [--lines N]] +# +# the Seqera run id (e.g. 3JG1eHsWrAeSse), as recorded in _runs.csv. +# --workspace Seqera workspace numeric id (default $TOWER_WORKSPACE_ID or 167877437119966). +# --stage override the output stage folder (else inferred from the swept token). +# --method override the swept method name used in the filename (else inferred). +# --base results-tree base (default $TXSIM_RESULTS_DIR or +# ~/projects/txsim_results/param_sweep_tests). +# --keep-json keep the raw tasks JSON next to the CSV. +# --diagnose pull the tail of the .command.err traceback off the scratch PVC (needs +# kubectl) for combos flagged for troubleshooting = methods that FAIL ON THEIR +# DEFAULT variant; saves to _run__failures/*.err. +# --all-failures with --diagnose, fetch tracebacks for EVERY failed combo, not just the +# fail-on-default ones (param-only failures included). +# --pod NAME pod (with tower-scratch mounted) to exec for --diagnose (default 'debug'). +# --lines N lines of .command.err to show/save per failure (default 40). +# +# Output: //_run__results.csv (+ summary + matrix; on failures also +# a *_failures.csv with a method-impl-vs-param `likely_class`, and, with --diagnose, +# the .command.err tracebacks). See SKILL.md "Classifying failures". +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +command -v tw >/dev/null 2>&1 || { echo "ERROR: 'tw' (Seqera CLI) not found on PATH"; exit 2; } +command -v jq >/dev/null 2>&1 || { echo "ERROR: 'jq' not found on PATH"; exit 2; } +command -v python3 >/dev/null 2>&1 || { echo "ERROR: 'python3' not found on PATH"; exit 2; } + +ID="" ; WS="${TOWER_WORKSPACE_ID:-167877437119966}" ; STAGE="" ; METHOD="" ; KEEP=0 +DIAGNOSE=0 ; POD="debug" ; LINES=40 ; ALLFAIL=0 +BASE="${TXSIM_RESULTS_DIR:-$HOME/projects/txsim_results/param_sweep_tests}" +while [ $# -gt 0 ]; do + case "$1" in + --workspace) WS="${2:-}"; shift 2;; + --stage) STAGE="${2:-}"; shift 2;; + --method) METHOD="${2:-}"; shift 2;; + --base) BASE="${2:-}"; shift 2;; + --keep-json) KEEP=1; shift;; + --diagnose) DIAGNOSE=1; shift;; + --all-failures) ALLFAIL=1; shift;; + --pod) POD="${2:-}"; shift 2;; + --lines) LINES="${2:-}"; shift 2;; + -h|--help) sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'; exit 0;; + -*) echo "unknown option: $1"; exit 2;; + *) if [ -z "$ID" ]; then ID="$1"; else echo "unexpected arg: $1"; exit 2; fi; shift;; + esac +done +[ -z "$ID" ] && { echo "usage: parse_run.sh [opts]"; exit 2; } +export TOWER_WORKSPACE_ID="$WS" + +tmp="$(mktemp -d "${TMPDIR:-/tmp}/parse_run.XXXXXX")" +trap '[ "$KEEP" -eq 1 ] || rm -rf "$tmp"' EXIT + +# ---- run-level headline ---- +status_txt="$(tw runs view -i "$ID" --status 2>&1)" || { echo "$status_txt"; echo "ERROR: could not fetch run $ID (auth? wrong id/workspace?)"; exit 1; } +overall="$(printf '%s\n' "$status_txt" | awk -F'|' '$1 ~ /[[:space:]]Status[[:space:]]/ {gsub(/^[ ]*|[ ]*$/,"",$2); print $2; exit}')" +run_name="$(printf '%s\n' "$status_txt" | awk -F'|' '$1 ~ /Run name/ {gsub(/^[ ]*|[ ]*$/,"",$2); print $2; exit}')" +[ -z "$overall" ] && overall="?" +[ -z "$run_name" ] && run_name="?" +echo "Fetching tasks for run $ID (status: $overall) ..." +case "$overall" in + SUCCEEDED|FAILED|CANCELLED|UNKNOWN|"?") : ;; + *) echo "NOTE: run is '$overall' — results may be incomplete (still running).";; +esac + +# ---- paginate all tasks (100/page cap) ---- +off=0 +while :; do + tw -o json runs view -i "$ID" tasks -c taskId,process,tag,status,exit --max 100 --offset "$off" \ + > "$tmp/page_$off.json" 2>"$tmp/page.err" \ + || { echo "ERROR fetching tasks at offset $off:"; cat "$tmp/page.err"; exit 1; } + n="$(jq 'length' "$tmp/page_$off.json" 2>/dev/null || echo 0)" + if [ "$n" -eq 0 ]; then rm -f "$tmp/page_$off.json"; break; fi + off=$((off + 100)) + [ "$n" -lt 100 ] && break +done +if ls "$tmp"/page_*.json >/dev/null 2>&1; then + jq -s 'add' "$tmp"/page_*.json > "$tmp/tasks.json" +else + echo "[]" > "$tmp/tasks.json" +fi +echo "tasks fetched: $(jq 'length' "$tmp/tasks.json")" +echo + +# ---- analyse ---- +extra=() +[ -n "$STAGE" ] && extra+=(--stage "$STAGE") +[ -n "$METHOD" ] && extra+=(--method "$METHOD") +py_out="$(python3 "$SCRIPT_DIR/parse_run.py" \ + --tasks "$tmp/tasks.json" --id "$ID" --base "$BASE" \ + --overall "$overall" --run-name "$run_name" \ + ${extra[@]+"${extra[@]}"} 2>&1)"; rc=$? +printf '%s\n' "$py_out" + +OUT_DIR="$(printf '%s\n' "$py_out" | sed -n 's/^OUT_DIR=//p' | tail -1)" +FT_FILE="$(printf '%s\n' "$py_out" | sed -n 's/^FAILED_TASKS=//p' | tail -1)" +CSV_FILE="$(printf '%s\n' "$py_out" | sed -n 's/^WROTE_CSV=//p' | tail -1)" + +# ---- --diagnose: pull .command.err tracebacks off scratch for flagged (fail-on-default) +# combos (or all failures with --all-failures) ---- +if [ "$DIAGNOSE" -eq 1 ] && [ -n "$FT_FILE" ] && [ -f "$FT_FILE" ]; then + if ! command -v kubectl >/dev/null 2>&1; then + echo; echo "WARN: --diagnose needs kubectl (not found) — skipping traceback fetch." + else + base="$(basename "${CSV_FILE:-run_${ID}_results.csv}" _results.csv)" + errdir="${OUT_DIR:-.}/${base}_failures" + mkdir -p "$errdir" + scope="fail-on-default only"; [ "$ALLFAIL" -eq 1 ] && scope="all failures" + echo; echo "=== DIAGNOSE ($scope): .command.err tail via pod '$POD' ===" + while IFS=$'\t' read -r tid ds combo step ex flag; do + [ -z "$tid" ] && continue + [ "$ALLFAIL" -eq 1 ] || [ "${flag:-0}" = "1" ] || continue # default: flagged only + wd="$(tw runs view -i "$ID" task -t "$tid" 2>/dev/null | awk -F'|' '/Work directory/{gsub(/^[ ]*|[ ]*$/,"",$2);print $2}')" + safe="$(printf '%s' "${ds}__${combo}" | tr ' /|=' '_____')" + errf="$errdir/${safe}.err" + echo; echo "----- $ds | $combo (step=$step exit=$ex taskId=$tid) -----" + if [ -z "$wd" ]; then echo " (no workdir from tw)"; continue; fi + if kubectl exec "$POD" -- sh -c "tail -n $LINES '$wd/.command.err' 2>/dev/null" > "$errf" 2>/dev/null && [ -s "$errf" ]; then + sed 's/^/ /' "$errf"; echo " (saved: $errf)" + else + echo " (could not read $wd/.command.err via pod '$POD')" + fi + done < "$FT_FILE" + fi +fi + +[ -n "$FT_FILE" ] && rm -f "$FT_FILE" 2>/dev/null # transient hand-off file +[ "$KEEP" -eq 1 ] && echo "(--keep-json) raw tasks JSON kept at: $tmp/tasks.json" +exit $rc diff --git a/.claude/skills/render-qmd-pod/SKILL.md b/.claude/skills/render-qmd-pod/SKILL.md new file mode 100644 index 000000000..d17a107ce --- /dev/null +++ b/.claude/skills/render-qmd-pod/SKILL.md @@ -0,0 +1,104 @@ +--- +name: render-qmd-pod +description: >- + Render a Quarto (.qmd) report against a given dataset-paths YAML on a pod on the + kubernetes cluster, using a Viash component's image (default process_dataset) so the + report's spatialdata/anndata/scanpy stack reads the combined datasets on the shared + scratch PVC natively. Use when you need to render/execute a .qmd (e.g. the dataset + visualisation report in scripts/results_report/) that reads data living ONLY on the + cluster's tower-scratch PVC — a local render can't see those files. Handles pod + creation, installing the render deps the image lacks (quarto, jupyter, spatialdata_plot), + staging the qmd+yaml, rendering, and copying the self-contained HTML back. +--- + +# render-qmd-pod + +The combined datasets (`/scratch/task_ist_preprocessing/datasets/…` + +`…/raw/…` single-cell refs) live **only** on the cluster's `tower-scratch` PVC, so a +report that reads them can't be rendered locally. This skill renders the `.qmd` on a +pod that **mounts that PVC** and runs a **component image** whose stack already reads +`.zarr`/`.h5ad` natively (default: `data_processors/process_dataset`, which ships +spatialdata 0.8 / anndata / scanpy / umap-learn / matplotlib). It then adds only the +report-rendering pieces the image lacks — **quarto**, **jupyter/ipykernel**, and any +report-specific python deps (**spatialdata_plot** by default) — stages the report, +renders, and copies the self-contained HTML back. + +The report/YAML contract (as in `scripts/results_report/Dataset_report.qmd`): the qmd +reads its paths from `$DATASET_CONFIG` (falling back to `dataset_paths.yaml`), resolved +relative to the qmd's own directory, and the YAML's `resources_roots` points at the PVC +mount. `render` sets `$DATASET_CONFIG` to the staged YAML's basename and runs quarto with +cwd = the qmd's dir, so paths resolve exactly as they do locally. + +## Prerequisites + +- `kubectl` pointed at the cluster (`kubectl config current-context`). On Nebius the + token expires — if a call prints "Switch to your browser to complete authentication", + re-auth before the next long transfer/exec. +- A **dataset-paths YAML** whose `resources_roots` points at the PVC mount + (`/scratch/task_ist_preprocessing`) and whose file paths exist there. To find the + smallest self-contained dataset (full SC ref + panel-subset SC ref + spatial zarr all + on the PVC), list `/scratch/task_ist_preprocessing/datasets` and `…/raw` on any pod + that mounts `/scratch` (e.g. the long-lived `debug` pod). `dataset_paths_skin_k8s.yaml` + is a worked example. +- The chosen component image must exist on ghcr (`check-component ` if unsure). + +## How to run + +```bash +S=.claude/skills/render-qmd-pod/render_qmd.sh +$S up # create pod (scratch mounted) + install render deps +$S render [out.html] # stage both, render, copy HTML back (default ./.html) +$S down # delete the pod when done +$S sh '' # poke the pod (debugging) +$S image # print the resolved image ref +``` + +Example (the dataset report on the smallest self-contained dataset): + +```bash +$S up +$S render scripts/results_report/Dataset_report.qmd \ + scripts/results_report/dataset_paths_skin_k8s.yaml \ + scripts/results_report/Dataset_report_skin_k8s.html +$S down +``` + +`up` is a one-time cost (~quarto 147MB download, cached on the PVC at +`/scratch/.render-cache`, + a pip install); after it, `render` reuses the pod and is +just the render itself. Re-render another dataset by pointing `render` at a different +YAML — no `up` needed. `up` is idempotent, so re-running it on an existing pod just +re-checks the deps. + +## Adapting + +- **Different image** (a report needing a different stack): `RENDER_COMPONENT=` + (resolved from `_viash.yaml` like the cluster pulls it) or `RENDER_IMAGE=`. +- **Extra python deps** the image lacks: `RENDER_PIP="pkg_a pkg_b"` (default + `spatialdata_plot`; the process_dataset image already has scanpy/spatialdata/umap-learn). +- **Other clusters**: `SCRATCH_PVC` / `SCRATCH_NS` / `SCRATCH_MOUNT` / `RENDER_POD` / + `QUARTO_VERSION` / `RENDER_KERNEL`. + +## Gotchas + +- **The YAML must point at the PVC, not a local checkout.** Its `resources_roots` first + entry must be the PVC mount (`/scratch/task_ist_preprocessing`); a local + `~/projects/.../resources_test` path won't exist in the pod. Keep the local path only as + a lower-priority fallback. +- **On-disk multiscale groups are `s0…s4`, not `scale0`.** spatialdata normalizes them to + `scale0` keys in memory, so a report accessing `img["scale0"]["image"]` works unchanged — + don't "fix" the report to `s0`. +- **`sleep infinity`, not a finite sleep** — the helper already does this; a completed pod + can't be exec'd. +- **Scratch PVC is namespace-scoped** (`tower-nf`), so the pod runs there and all + exec/cp/down use that namespace automatically. +- **If a render exec drops mid-stream**, the pod persists and the working copy + `.html` + stay in `/tmp/render-qmd` inside the pod — re-run `render` (cheap once deps are installed) + or fetch the file with `$S sh 'ls -la /tmp/render-qmd'` + `kubectl cp`. +- **Leaving the pod up** holds a PVC mount + a node slot. Fast to re-render, but run + `$S down` when finished. + +## After you render + +- Open the HTML locally (`open `) to check the figures. +- The self-contained HTML embeds all figures (base64 PNGs), so it's shareable as-is. +- `$S down` to release the pod. diff --git a/.claude/skills/render-qmd-pod/render_qmd.sh b/.claude/skills/render-qmd-pod/render_qmd.sh new file mode 100644 index 000000000..9125448db --- /dev/null +++ b/.claude/skills/render-qmd-pod/render_qmd.sh @@ -0,0 +1,219 @@ +#!/usr/bin/env bash +# render_qmd.sh — render a Quarto .qmd report against a given dataset-paths YAML on +# a pod on the CURRENT kubectl cluster, using a Viash component's container image so +# the report's spatial/single-cell stack (spatialdata, anndata, scanpy, ...) reads +# the data on the shared scratch PVC natively — no ad-hoc reader installs. +# +# Why a pod: the combined datasets live ONLY on the cluster's scratch PVC +# (tower-scratch, ~/scratch/task_ist_preprocessing/...). A local render can't see +# them. So we mount that PVC into a pod running the process_dataset image (which +# already has spatialdata 0.8 / anndata / scanpy / umap-learn / matplotlib) and add +# just the report-rendering pieces the image lacks: quarto, jupyter/ipykernel, and +# any report-specific python deps (spatialdata_plot by default). +# +# Usage: +# render_qmd.sh up # create pod (scratch mounted) + install render deps +# render_qmd.sh render [out.html] +# # stage both into the pod, quarto render, copy HTML back +# render_qmd.sh down # delete the pod +# render_qmd.sh sh '' # run a bash snippet in the pod (debugging) +# render_qmd.sh image # print the resolved image ref +# +# The report reads its config via $DATASET_CONFIG (falls back to dataset_paths.yaml); +# `render` sets it to the staged YAML's basename and runs quarto with cwd = the qmd's +# dir, so the YAML's relative paths resolve exactly as they do locally. The YAML's +# resources_roots must point at the PVC mount (default /scratch/task_ist_preprocessing). +# +# Env overrides (defaults target this repo's Nebius/Seqera setup): +# RENDER_POD (render-qmd) SCRATCH_PVC (tower-scratch) SCRATCH_NS (tower-nf) +# SCRATCH_MOUNT (/scratch) RENDER_COMPONENT (data_processors/process_dataset) +# RENDER_TAG (build_main) RENDER_IMAGE (full ref; overrides component/tag) +# QUARTO_VERSION (1.10.18) RENDER_PIP (spatialdata_plot) RENDER_KERNEL (python3) +set -uo pipefail + +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)" || { echo "ERROR: not in a git repo"; exit 2; } +cd "$REPO_ROOT" || exit 2 + +CMD="${1:-}" +[ -z "$CMD" ] && { grep -E '^#( |$)' "$0" | sed -E 's/^# ?//'; exit 2; } + +# ---- config (env-overridable) ---- +POD="${RENDER_POD:-render-qmd}" +NS="${SCRATCH_NS:-tower-nf}" +PVC="${SCRATCH_PVC:-tower-scratch}" +MOUNT="${SCRATCH_MOUNT:-/scratch}" +QVER="${QUARTO_VERSION:-1.10.18}" +KERNEL="${RENDER_KERNEL:-python3}" +# report-specific python deps the base image lacks (jupyter/ipykernel/quarto are always added) +RENDER_PIP="${RENDER_PIP:-spatialdata_plot}" + +# ---- image ref from _viash.yaml (same resolution the cluster pulls with) ---- +ident() { grep -E "$1" _viash.yaml | head -1 | sed -E 's/.*:[[:space:]]*//; s/[[:space:]]*$//'; } +ORG="$(ident '^organization:')"; PROJ="$(ident '^name:')" +REG="$(ident 'docker_registry:')"; REG="${REG:-ghcr.io}" +COMPONENT="${RENDER_COMPONENT:-data_processors/process_dataset}" +TAG="${RENDER_TAG:-build_main}" +IMAGE="${RENDER_IMAGE:-$REG/$ORG/$PROJ/$COMPONENT:$TAG}" + +CACHE="$MOUNT/.render-cache" # quarto cached here on the PVC so re-setup skips the 147MB download +WORK="/tmp/render-qmd" # per-render staging dir inside the pod (kept off the shared PVC) + +xp() { kubectl exec -n "$NS" "$POD" -- bash -c "$1"; } # exec a bash snippet in the pod + +case "$CMD" in + image) echo "$IMAGE" ;; + + up) + echo "Pod: $POD (namespace $NS)" + echo "Image: $IMAGE" + echo "Mount: PVC $PVC -> $MOUNT" + if kubectl get pod "$POD" -n "$NS" >/dev/null 2>&1; then + echo "pod already exists — (re)installing render deps (idempotent)" + else + # kubectl run can't attach a PVC volume -> apply a manifest. `sleep infinity` + # so the pod never completes (you can't exec a completed pod). + cat <&1 | tail -1 +apiVersion: v1 +kind: Pod +metadata: + name: $POD + namespace: $NS +spec: + restartPolicy: Never + containers: + - name: main + image: $IMAGE + command: ["sleep", "infinity"] + volumeMounts: + - name: scratch + mountPath: $MOUNT + volumes: + - name: scratch + persistentVolumeClaim: + claimName: $PVC +YAML + echo "waiting for Ready (image may be multi-GB; cached nodes are instant)..." + kubectl wait --for=condition=Ready "pod/$POD" -n "$NS" --timeout=600s 2>&1 | tail -1 + fi + + echo "installing render deps (quarto $QVER + jupyter + $KERNEL kernel + pip: $RENDER_PIP)..." + xp ' + set -e + QVER="'"$QVER"'"; CACHE="'"$CACHE"'"; KERNEL="'"$KERNEL"'" + QDIR="$CACHE/quarto-$QVER" + # quarto: cache the tarball extraction on the PVC, symlink into PATH (idempotent) + if [ ! -x "$QDIR/bin/quarto" ]; then + echo " downloading quarto $QVER ..." + mkdir -p "$CACHE" + URL="https://github.com/quarto-dev/quarto-cli/releases/download/v$QVER/quarto-$QVER-linux-amd64.tar.gz" + curl -sSL "$URL" -o /tmp/quarto.tgz + tar -xzf /tmp/quarto.tgz -C "$CACHE" + rm -f /tmp/quarto.tgz + else + echo " quarto $QVER already cached at $QDIR" + fi + ln -sf "$QDIR/bin/quarto" /usr/local/bin/quarto + # quarto jupyter engine + report-specific python deps (pip is idempotent) + echo " pip install jupyter ipykernel nbclient '"$RENDER_PIP"' ..." + pip install --quiet --no-input --root-user-action=ignore jupyter ipykernel nbclient '"$RENDER_PIP"' + python -m ipykernel install --sys-prefix --name "$KERNEL" --display-name "$KERNEL" >/dev/null 2>&1 + echo " quarto: $(quarto --version) kernel: $KERNEL registered" + ' + ;; + + render) + QMD="${2:?need a .qmd file as arg 2}" + YAML="${3:?need a dataset-paths .yaml as arg 3}" + OUT="${4:-}" + [ -f "$QMD" ] || { echo "ERROR: qmd not found: $QMD"; exit 2; } + [ -f "$YAML" ] || { echo "ERROR: yaml not found: $YAML"; exit 2; } + kubectl get pod "$POD" -n "$NS" >/dev/null 2>&1 || { echo "ERROR: pod $POD not up — run: $0 up"; exit 2; } + + QBASE="$(basename "$QMD")"; YBASE="$(basename "$YAML")" + STEM="${QBASE%.*}" + OUT="${OUT:-./$STEM.html}" + + # Per-dataset work dir keyed by the yaml name, so re-rendering another dataset on + # the same pod never picks up a previous render's leftovers, and so a resume + # targets the right render. (Renders on one pod are sequential per caller.) + RSLUG="$(printf '%s' "$YBASE" | tr -c 'A-Za-z0-9_.-' '_')" + RWORK="$WORK/$RSLUG" + RFLAG="$RWORK/render.rc"; RLOG="$RWORK/render.log"; RPID="$RWORK/render.pid" + RUN="$RWORK/.render_run.sh"; HTML="$RWORK/$STEM.html" + + # Whole-section transcript scatters (5-30M points) run for minutes with no stream + # output, long enough for the kubectl-exec websocket to reset and kill a synchronous + # exec mid-render. So run quarto DETACHED (nohup, survives the drop) and poll with + # short execs. The render is RESUMABLE: if this call is interrupted (e.g. a client + # timeout), just run the SAME command again — it re-attaches to the running render + # (or copies the finished HTML) instead of restarting it. + # done = finished OK (rc 0 + html present); running = live pid; none = never ran, + # still running-but-no-pid, OR a PRIOR FAILED run (rc != 0 / no html) -> relaunch fresh. + _state="$(xp "if [ -f '$RFLAG' ]; then if [ \"\$(cat '$RFLAG' 2>/dev/null)\" = 0 ] && [ -f '$HTML' ]; then echo done; else echo none; fi; elif [ -f '$RPID' ] && kill -0 \$(cat '$RPID') 2>/dev/null; then echo running; else echo none; fi" 2>/dev/null | tr -d '[:space:]')" + + if [ "$_state" = none ]; then + echo "staging $QBASE + $YBASE into $POD:$RWORK ..." + xp "rm -rf '$RWORK' && mkdir -p '$RWORK'" + kubectl cp "$QMD" "$NS/$POD:$RWORK/$QBASE" 2>&1 | grep -v 'Removing leading' || true + kubectl cp "$YAML" "$NS/$POD:$RWORK/$YBASE" 2>&1 | grep -v 'Removing leading' || true + # launcher writes quarto's own rc AFTER it exits, so $RFLAG's presence == done. + xp "cat > '$RUN' <<'SH' +cd '$RWORK' +DATASET_CONFIG='$YBASE' MPLBACKEND=Agg quarto render '$QBASE' --to html +echo \$? > '$RFLAG' +SH" + echo "launching detached render (nohup; survives exec/websocket drops) ..." + xp "cd '$RWORK' && rm -f '$RFLAG' '$HTML' && nohup bash '$RUN' '$RLOG' 2>&1 & echo \$! > '$RPID'" + elif [ "$_state" = running ]; then + echo "a render for $YBASE is already in progress in $POD — resuming poll (not restarting) ..." + else + echo "render for $YBASE already complete in $POD — copying back ..." + fi + + if [ "$_state" != done ]; then + echo "polling (whole-section renders can take several min; interruptible + resumable) ..." + RC=""; _last="" + for _i in $(seq 1 300); do # ceiling; a client timeout just means re-run + sleep 12 + RC="$(xp "cat '$RFLAG' 2>/dev/null" 2>/dev/null | tr -dc '0-9-')" + _prog="$(xp "grep -oE 'Cell [0-9]+/[0-9]+|Output created' '$RLOG' 2>/dev/null | tail -n1" 2>/dev/null)" + [ -n "$_prog" ] && [ "$_prog" != "$_last" ] && { echo " $_prog"; _last="$_prog"; } + [ -n "$RC" ] && break + _alive="$(xp "[ -f '$RPID' ] && kill -0 \$(cat '$RPID') 2>/dev/null && echo y || echo n" 2>/dev/null | tr -d '[:space:]')" + if [ "$_alive" = n ]; then + sleep 3 + RC="$(xp "cat '$RFLAG' 2>/dev/null" 2>/dev/null | tr -dc '0-9-')" + [ -n "$RC" ] && break + echo "ERROR: render process exited without writing rc (likely OOM/kill). Log tail:" + xp "tail -n 25 '$RLOG'" 2>/dev/null + exit 4 + fi + done + [ -z "$RC" ] && { echo "still rendering after poll ceiling — resumable: re-run the same command. Log: $POD:$RLOG"; exit 5; } + if [ "$RC" -ne 0 ]; then + echo "ERROR: quarto render failed (rc=$RC). Log tail:" + xp "tail -n 25 '$RLOG'" 2>/dev/null + exit "$RC" + fi + fi + + echo "copying HTML back to $OUT ..." + kubectl cp "$NS/$POD:$HTML" "$OUT" 2>&1 | grep -v 'Removing leading' || true + if [ -f "$OUT" ]; then + echo "OK: $OUT ($(du -h "$OUT" | cut -f1))" + else + echo "ERROR: render reported success but $STEM.html not found in pod; check $POD:$RWORK" + exit 3 + fi + ;; + + down) + kubectl delete pod "$POD" -n "$NS" --wait=false 2>&1 + ;; + + sh) + xp "${2:?need a bash snippet as arg 2}" + ;; + + *) echo "unknown command '$CMD' (up|render|down|sh|image)"; exit 2 ;; +esac diff --git a/.claude/skills/restore-pipeline/SKILL.md b/.claude/skills/restore-pipeline/SKILL.md new file mode 100644 index 000000000..8446b76b2 --- /dev/null +++ b/.claude/skills/restore-pipeline/SKILL.md @@ -0,0 +1,135 @@ +--- +name: restore-pipeline +description: >- + Resume a FROZEN or CANCELLED Seqera/Nextflow benchmark run on the Nebius k8s env WITHOUT + re-running the tasks it already completed — by preserving its -resume cache and injecting it + into a fresh head via a --pre-run script (Tower's own relaunch does NOT restore the cache, so + a plain relaunch re-runs everything from scratch). Use when a run's head froze (JVM hung — + still "RUNNING" but silent for hours, submitting nothing) or was cancelled, and you want to + recover it; also for relaunching a run with a method dropped or a changed retry/memory policy + while keeping the cache. Complements fetch-run-results (which copies a finished run's outputs). +--- + +# restore-pipeline + +Resume a run that **froze** or was **cancelled**, reusing everything it already computed. On this +Nebius compute env a plain `tw runs relaunch --resume` re-runs **from scratch** — this skill makes +it a real resume (validated: **2317/2420 tasks Cached** on `grave_carson`, `~1031/1061` on +`nauseous_celsius`). + +Background + full war story: memory **[[nextflow-resume-cache-on-ephemeral-head-pod]]** and +**[[tw-resume-git-commit-cache-bust]]**. + +## When to use + +- A run's **head froze**: Tower still shows `RUNNING`, but the head pod's log is silent for hours, + `RESTARTS 0`, and it has **submitted no new task pods** since the freeze (a hung JVM — often GC + death-spiral under `0/76 nodes: Insufficient memory/ephemeral-storage`). It will never finish. +- A run was **cancelled** and you want to continue it. +- You want to **relaunch with a change** (drop a buggy method, change retry/memory) but keep the + cache. + +## The core problem (why a plain relaunch fails) + +Nextflow reads its `-resume` cache DB (`.nextflow/cache//` = LevelDB `db/` + +`index.`) from the **head pod's LAUNCH dir** (`/` -> `/.nextflow/cache`), **not** from the +workDir on `/scratch`. `tw runs relaunch` passes `-resume ` and reuses the workDir, but +starts the new head with an **empty** `/.nextflow/cache` and never restores the flushed cache → +`0 Cached / N Submitted` (verified). The fix: a `--pre-run` script copies the real cache into +`/.nextflow/cache` **before** Nextflow starts. + +**Where the cache lives depends on state** (the script handles both): +- **RUNNING / FROZEN** → only on the head pod's **ephemeral** fs (`/.nextflow/cache/`). + If the pod is evicted or cancelled ungracefully, it is **LOST**. Back it up FIRST. +- **CANCELLED / finalized** → Tower flushed it to `/.nextflow/cache/` on + `/scratch`. Still back it up (insurance). + +## How to run + +```bash +.claude/skills/restore-pipeline/restore_pipeline.sh [opts] +``` + +**Frozen run — do the urgent backup first, then resume:** +```bash +# 1) IMMEDIATELY preserve the cache off the live (frozen) head pod, before anything kills it: +.claude/skills/restore-pipeline/restore_pipeline.sh --backup-only +# 2) resume (cancels the frozen run, injects the cache, pins the exact commit): +.claude/skills/restore-pipeline/restore_pipeline.sh --cancel +``` + +**Cancelled run — one shot:** +```bash +.claude/skills/restore-pipeline/restore_pipeline.sh +``` + +**With modifications (both cache-safe — see below):** +```bash +.claude/skills/restore-pipeline/restore_pipeline.sh --cancel \ + --config resume_override.config \ # fail-fast retry/memory (template provided) + --params-file my_params.yaml # e.g. a method dropped +``` + +Key options: `--backup-only`, `--cancel`, `--commit `, `--session `, `--config `, +`--params-file `, `--name`, `--dry-run`, `--workspace`, `--pod`. Defaults for `--commit` / +`--session` come from `tw runs view -i ` (the run's own Commit ID / Session ID). + +## What the script does + +1. Resolve `session`, `commit`, `status`, `workDir` from `tw runs view`. +2. **Back up the cache** to `/scratch/resume_backup//cache/` — from the live head + pod if running, else from the flushed copy on `/scratch`. +3. (`--cancel`) cancel the live run. +4. Generate a `--pre-run` injector (subshell; **no `set -e/-u`** — `set -u` leaks into + `nf-launcher.sh` and kills it on unbound `NXF_XPACK_LICENSE`). +5. `tw runs relaunch -i --revision --pre-run [--config …] [--params-file …]`. +6. Print the verification commands. + +## Two cache-safe modifications you can fold into the resume + +Nextflow hashes a task's *identity* (script, inputs, container, params it consumes, `git_commit`), +**not** resource/execution directives or unrelated branches — so these don't bust the cache: + +- **Drop a method** → build a replacement `--params-file` from `tw runs view -i --params` + *verbatim*, removing the method from its `_methods` array (the method lists live inside a + JSON-string `settings` key; the stage keys are real workflow params — `--transcript_assignment_methods` + etc.). The removed branch generates no tasks; kept branches stay Cached. (Include-list only: + `checkItemAllowed(item, include=state[key], exclude=null)` — to drop one, list all-except-it.) +- **Change retry/memory** → `--config resume_override.config` (copy the provided + `resume_override.config.template`). Fail-fast (`errorStrategy='ignore'; maxRetries=0`) stops OOM + tasks escalating to 480 GB and hogging nodes. Mind the two footguns documented in the template + (`get_memory` forcing 480 GB when `attempt==maxRetries`; `exitStrat` halting on too-few retries). + +## Verify (always) + +Once the new head schedules and starts: +```bash +H=$(kubectl get pods --no-headers | grep | grep workflow | awk '{print $1}') +kubectl logs "$H" | grep -i '\[pre-run\]' # cache was injected +kubectl logs "$H" | grep -oE 'Cached process|Submitted process' | sort | uniq -c # want mostly Cached +``` +Mostly `Cached process` → success. `0 Cached` → the injection or the commit pin failed (see gotchas). + +## Gotchas + +- **Pin the EXACT original commit.** `--revision ` (the script defaults to the run's own + Commit ID). A newer `build/main` re-stamps `meta.git_commit` into every module → 100% hash miss. + `--commit-id` does **not** pin — pass the SHA as the revision. See [[tw-resume-git-commit-cache-bust]]. +- **In-flight tasks are unrecoverable.** A frozen head can't record a task's completion in the + cache, so any task still running at freeze time re-runs on resume (its on-disk output isn't in + the cache DB). Waiting for them is pointless — resume now under a live head that *will* record them. +- **The pre-run must not `set -e/-u`.** It is sourced into `nf-launcher.sh`; `set -u` kills it on + unbound `NXF_XPACK_LICENSE`. The generated script uses a subshell and never fails the launcher. +- **Capacity.** These runs have huge per-task memory (up to 480 GiB ≈ a whole node). Don't resume + many at once, and consider the fail-fast `--config` to cap memory (300 GiB) so tails schedule. +- **Auth/context.** `tw` authenticated to the workspace and `kubectl` on the nebius context + (`kubectl config current-context` → `nebius-mk8s-…`). Nebius tokens expire; re-auth if a call + prints "Switch to your browser to complete authentication". +- **macOS bash 3.2** has no `declare -A` — any helper loops you write around this must use temp + files or plain vars, not associative arrays. + +## Output + +Report: the resolved session/commit, where the cache was backed up (and whether from the live head +or scratch — critical for a frozen run), the new run id, and the Cached/Submitted verification. +Then hold or proceed per the user's plan (e.g. stagger multiple resumes for capacity). diff --git a/.claude/skills/restore-pipeline/restore_pipeline.sh b/.claude/skills/restore-pipeline/restore_pipeline.sh new file mode 100644 index 000000000..6ad76aadc --- /dev/null +++ b/.claude/skills/restore-pipeline/restore_pipeline.sh @@ -0,0 +1,216 @@ +#!/usr/bin/env bash +# restore_pipeline.sh — resume a FROZEN or CANCELLED Seqera/Nextflow benchmark run +# on the Nebius k8s env WITHOUT re-running the tasks it already completed. +# +# WHY THIS EXISTS (the core trick): +# Nextflow reads its -resume cache DB from the head pod's LAUNCH dir (/.nextflow/cache/), +# NOT from the workDir on /scratch. `tw runs relaunch` DOES pass `-resume ` and reuse +# the workDir, but it starts the new head with an EMPTY /.nextflow/cache and never restores the +# flushed cache -> so a plain relaunch re-runs EVERYTHING from scratch (verified: 0 Cached). +# The fix is a `--pre-run` script that copies the real cache into /.nextflow/cache BEFORE +# Nextflow starts. That + pinning the EXACT original commit (so task hashes match) gives a real +# resume (verified: 2317/2420 Cached on grave_carson). +# +# Where the cache lives depends on run state: +# - RUNNING/FROZEN run -> ONLY on the head pod's ephemeral fs at /.nextflow/cache/. +# If the pod is evicted/cancelled ungracefully it is LOST. Back it up +# FIRST (this script does), then cancel. +# - CANCELLED/finalized -> Tower already flushed it to /.nextflow/cache/ on +# /scratch. Still back it up (insurance) before touching anything. +# +# Usage: +# restore_pipeline.sh [opts] +# +# --backup-only Only preserve the cache to /scratch/resume_backup// and stop. +# (Do this IMMEDIATELY for a frozen run, before it is cancelled/evicted.) +# --cancel If the run is still live (RUNNING/SUBMITTED), cancel it after backing up +# the cache and before relaunching. (In-flight tasks are lost either way — +# a frozen head can't cache their completion.) +# --commit Override the revision to relaunch on. Default: the run's own "Commit ID" +# (tw runs view). MUST be the exact original commit or every task hash +# misses (viash re-stamps meta.git_commit). Passed as --revision (NOT +# --commit-id, which does not pin). +# --session Override the Nextflow session id. Default: the run's "Session ID". +# --config Extra Nextflow config layered onto the relaunch (e.g. the fail-fast +# retry/memory override — see resume_override.config.template). CACHE-SAFE: +# resource/execution directives are not part of the task hash. +# --params-file Replacement params-file (e.g. to drop a method from a _methods +# list). Build it from `tw runs view -i --params` verbatim, changing +# only what you intend. CACHE-SAFE for the branches you keep. +# --name Run name for the relaunch (default: _resume). +# --dry-run Print the relaunch command (and generated pre-run) without firing it. +# --workspace Seqera workspace (default $TOWER_WORKSPACE_ID or 167877437119966). +# --pod Preferred pod mounting the scratch PVC (default $SCRATCH_POD or 'debug'). +# If it's absent, the script AUTO-RESOLVES: it uses any Running pod that +# mounts the PVC (prefers an nf-workflow head), else spawns a short-lived +# pod that mounts the PVC and tears it down on exit. So 'debug' need not exist. +# --namespace Namespace of the scratch pod / PVC (default $SCRATCH_NS or 'tower-nf'). +# --pvc Scratch PVC name to mount if a pod must be spawned (default $SCRATCH_PVC +# or 'tower-scratch'). +# +# Env: TOWER_WORKSPACE_ID, SCRATCH_POD, SCRATCH_NS, SCRATCH_PVC, SCRATCH_MOUNT(/scratch), +# BACKUP_ROOT(/scratch/resume_backup), RESTORE_IMAGE(ephemeral-pod image) +set -uo pipefail + +command -v tw >/dev/null 2>&1 || { echo "ERROR: 'tw' (Seqera CLI) not found on PATH"; exit 2; } +command -v kubectl >/dev/null 2>&1 || { echo "ERROR: 'kubectl' not found on PATH"; exit 2; } + +RUN="" ; BACKUP_ONLY=0 ; DO_CANCEL=0 ; COMMIT="" ; SESSION="" ; CONFIG="" ; PARAMS="" ; NAME="" ; DRY=0 +WS="${TOWER_WORKSPACE_ID:-167877437119966}" +POD="${SCRATCH_POD:-debug}" +NS="${SCRATCH_NS:-tower-nf}" +PVC="${SCRATCH_PVC:-tower-scratch}" +MOUNT="${SCRATCH_MOUNT:-/scratch}" +BACKUP_ROOT="${BACKUP_ROOT:-/scratch/resume_backup}" + +while [ $# -gt 0 ]; do + case "$1" in + --backup-only) BACKUP_ONLY=1;; + --cancel) DO_CANCEL=1;; + --commit) COMMIT="$2"; shift;; + --session) SESSION="$2"; shift;; + --config) CONFIG="$2"; shift;; + --params-file) PARAMS="$2"; shift;; + --name) NAME="$2"; shift;; + --dry-run) DRY=1;; + --workspace) WS="$2"; shift;; + --pod) POD="$2"; shift;; + --namespace) NS="$2"; shift;; + --pvc) PVC="$2"; shift;; + -h|--help) sed -n '2,51p' "$0"; exit 0;; + -*) echo "unknown option: $1"; exit 2;; + *) [ -z "$RUN" ] && RUN="$1" || { echo "unexpected arg: $1"; exit 2; };; + esac + shift +done +[ -n "$RUN" ] || { echo "ERROR: run_id required. See --help."; exit 2; } +export TOWER_WORKSPACE_ID="$WS" + +echo ">>> Resolving run $RUN ..." +VIEW="$(tw runs view -i "$RUN" 2>&1)" || { echo "$VIEW"; echo "ERROR: tw runs view failed (auth? bad id?)"; exit 3; } +get(){ echo "$VIEW" | awk -F'|' "/ $1 /{gsub(/^ +| +\$/,\"\",\$2);print \$2; exit}"; } +STATUS="$(get Status)"; ONAME="$(get 'Run name')"; WORKDIR="$(get Workdir)" +[ -n "$COMMIT" ] || COMMIT="$(get 'Commit ID')" +[ -n "$SESSION" ] || SESSION="$(get 'Session ID')" +[ -n "$NAME" ] || NAME="${ONAME}_resume" +echo " name=$ONAME status=$STATUS" +echo " session=$SESSION commit=$COMMIT workdir=$WORKDIR" +[ -n "$SESSION" ] && [ -n "$COMMIT" ] || { echo "ERROR: could not resolve session/commit"; exit 3; } + +DEST="$BACKUP_ROOT/$RUN/cache" +LC="$(echo "$RUN" | tr 'A-Z' 'a-z')" +HEAD="$(kubectl get pods -n "$NS" --no-headers 2>/dev/null | grep "$LC" | grep workflow | awk '{print $1; exit}')" + +# Resolve a pod that mounts the scratch PVC (for reading/writing /scratch). Do NOT +# hard-depend on a 'debug' pod that may not exist: prefer $POD if it's up, else ANY +# Running pod that mounts $PVC (usually a live nf-workflow head), else spawn an +# ephemeral pod that mounts $PVC and tear it down on exit. +SPAWNED=0 +find_pvc_pod() { + kubectl get pods -n "$NS" -o json 2>/dev/null | python3 -c ' +import json,sys +pvc=sys.argv[1] +best=None +for p in json.load(sys.stdin).get("items",[]): + if p.get("status",{}).get("phase")!="Running": continue + for v in p["spec"].get("volumes",[]): + if ((v.get("persistentVolumeClaim") or {}).get("claimName"))==pvc: + nm=p["metadata"]["name"] + # prefer a workflow head pod (stable), else take the first match + if nm.startswith("nf-workflow-"): print(nm); sys.exit(0) + best=best or nm +if best: print(best) +' "$PVC" 2>/dev/null +} +resolve_scratch_pod() { + if [ "$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.status.phase}' 2>/dev/null)" = Running ]; then + SPOD="$POD"; echo " scratch pod: existing '$SPOD'"; return 0 + fi + SPOD="$(find_pvc_pod)" + if [ -n "$SPOD" ]; then echo " scratch pod: '$SPOD' (mounts $PVC)"; return 0; fi + # spawn ephemeral — image ref from _viash.yaml (guaranteed-pullable, like fetch-run-results) + SPOD="restore-scratch-$LC" + ident() { grep -E "$1" _viash.yaml 2>/dev/null | head -1 | sed -E 's/.*:[[:space:]]*//; s/[[:space:]]*$//'; } + local ORG PROJ REG IMG + ORG="$(ident '^organization:')"; PROJ="$(ident '^name:')"; REG="$(ident 'docker_registry:')"; REG="${REG:-ghcr.io}" + IMG="${RESTORE_IMAGE:-$REG/$ORG/$PROJ/data_processors/process_dataset:build_main}" + echo " no scratch pod found — spawning ephemeral '$SPOD' (image $IMG) ..." + cat </dev/null 2>&1 +apiVersion: v1 +kind: Pod +metadata: { name: $SPOD, namespace: $NS } +spec: + restartPolicy: Never + containers: + - { name: main, image: $IMG, command: ["sleep","infinity"], volumeMounts: [{ name: scratch, mountPath: $MOUNT }] } + volumes: + - { name: scratch, persistentVolumeClaim: { claimName: $PVC } } +YAML + SPAWNED=1 + kubectl wait --for=condition=Ready "pod/$SPOD" -n "$NS" --timeout=600s >/dev/null 2>&1 \ + || { echo "ERROR: ephemeral scratch pod not Ready"; exit 4; } +} +cleanup_spod() { [ "${SPAWNED:-0}" -eq 1 ] && kubectl delete pod "$SPOD" -n "$NS" --wait=false >/dev/null 2>&1; } +trap cleanup_spod EXIT + +echo ">>> Backing up cache for session $SESSION -> $DEST/$SESSION" +if [ -n "$HEAD" ] && kubectl exec -n "$NS" "$HEAD" -- test -d "/.nextflow/cache/$SESSION" 2>/dev/null; then + echo " source: LIVE head pod $HEAD (/.nextflow/cache) — this is the only copy for a running run" + kubectl exec -n "$NS" "$HEAD" -- sh -c "mkdir -p '$DEST' && cp -a '/.nextflow/cache/$SESSION' '$DEST/' && du -sh '$DEST/$SESSION'" \ + || { echo "ERROR: cache backup from head pod failed"; exit 4; } +else + resolve_scratch_pod + if kubectl exec -n "$NS" "$SPOD" -- test -d "$WORKDIR/.nextflow/cache/$SESSION" 2>/dev/null; then + echo " source: flushed cache on scratch ($WORKDIR/.nextflow/cache) via pod $SPOD" + kubectl exec -n "$NS" "$SPOD" -- sh -c "mkdir -p '$DEST' && cp -a '$WORKDIR/.nextflow/cache/$SESSION' '$DEST/' && du -sh '$DEST/$SESSION'" \ + || { echo "ERROR: cache backup from scratch failed"; exit 4; } + elif kubectl exec -n "$NS" "$SPOD" -- test -d "$DEST/$SESSION" 2>/dev/null; then + echo " backup already exists at $DEST/$SESSION (reusing)" + kubectl exec -n "$NS" "$SPOD" -- du -sh "$DEST/$SESSION" 2>/dev/null + else + echo "ERROR: cache DB not found on head pod OR in $WORKDIR/.nextflow/cache OR in $DEST." + echo " If the head pod was already evicted/cancelled ungracefully the cache is lost." + exit 4 + fi +fi + +if [ "$BACKUP_ONLY" -eq 1 ]; then + echo ">>> --backup-only: cache preserved at $DEST/$SESSION. Stopping." + exit 0 +fi + +if [ "$DO_CANCEL" -eq 1 ] && ! echo "$STATUS" | grep -qE 'SUCCEEDED|FAILED|CANCELLED|UNKNOWN'; then + echo ">>> Cancelling live run $RUN (in-flight tasks are unrecoverable for resume anyway) ..." + [ "$DRY" -eq 1 ] && echo " (dry-run) tw runs cancel -i $RUN" || tw runs cancel -i "$RUN" +fi + +# Generate the pre-run cache-injection script. SOURCED into nf-launcher.sh, so a subshell and +# NO `set -e/-u` (set -u leaks and kills the launcher on unbound NXF_XPACK_LICENSE). +PRERUN="$(mktemp -t prerun_inject_"$RUN"_XXXX.sh)" +cat > "$PRERUN" < \$DST" + mkdir -p "\$DST"; rm -rf "\$DST/\$SESS"; cp -a "\$SRC" "\$DST/" + echo "[pre-run] injected: \$(du -sh "\$DST/\$SESS" | cut -f1)" ) \\ + || echo "[pre-run] injection failed (continuing without resume cache)" +EOF +echo ">>> Generated pre-run injector: $PRERUN" + +set -- -i "$RUN" --revision "$COMMIT" --name "$NAME" --pre-run "$PRERUN" +[ -n "$CONFIG" ] && set -- "$@" --config "$CONFIG" +[ -n "$PARAMS" ] && set -- "$@" --params-file "$PARAMS" + +echo ">>> Relaunch command:" +echo " tw runs relaunch $*" +if [ "$DRY" -eq 1 ]; then echo ">>> --dry-run: not firing."; exit 0; fi + +OUT="$(tw runs relaunch "$@" 2>&1)"; echo "$OUT" +NEWID="$(echo "$OUT" | grep -oE 'Workflow [A-Za-z0-9]+' | awk '{print $2; exit}')" +echo +echo ">>> Verify cache reuse once the new head starts (want mostly 'Cached process'):" +echo " H=\$(kubectl get pods --no-headers | grep \$(echo ${NEWID:-} | tr A-Z a-z) | grep workflow | awk '{print \$1}')" +echo " kubectl logs \"\$H\" | grep -oE 'Cached process|Submitted process' | sort | uniq -c" +echo " kubectl logs \"\$H\" | grep -i '\\[pre-run\\]' # confirm the cache was injected" diff --git a/.claude/skills/restore-pipeline/resume_override.config.template b/.claude/skills/restore-pipeline/resume_override.config.template new file mode 100644 index 000000000..8d30f5521 --- /dev/null +++ b/.claude/skills/restore-pipeline/resume_override.config.template @@ -0,0 +1,37 @@ +// resume_override.config — OPTIONAL fail-fast retry/memory override for a resume. +// Pass to restore_pipeline.sh with `--config`. CACHE-SAFE: Nextflow does not hash +// resource/execution directives, so this only affects the re-running tail; the +// resumed (Cached) tasks are untouched. Layered LAST, so it overrides +// src/base/labels_nebius.config. Edit the numbers to taste before using. +// +// Two footguns in labels_nebius.config this file sidesteps: +// 1. get_memory() returns maxMemory (480GB) whenever task.attempt == maxRetries, +// so a naive `maxRetries=1` forces 480GB on attempt 1. -> set maxMemory=null +// AND give each label a FLAT memory (no `* task.attempt`). +// 2. exitStrat() hardcodes `attempt>=3 -> ignore`; with fewer retries it can +// return 'retry' with no retries left and HALT the workflow. -> override +// errorStrategy explicitly. +process { + // 0-retry fail-fast: one attempt, on any failure skip the combo and continue. + // (A task that OOMs at flat memory is dropped, not escalated to 480GB. A few + // method×dataset cells may be absent from the final scores — accepted tradeoff.) + errorStrategy = 'ignore' + maxRetries = 0 + maxMemory = null + + // Flat memory so nothing requests the 480 GiB node-filling amount. + withLabel: lowmem { memory = 25.GB } + withLabel: midmem { memory = 50.GB } + withLabel: highmem { memory = 200.GB } + withLabel: veryhighmem { memory = 300.GB } // base is 480 in labels_nebius.config + + // Exception: give a metric you WANT to complete headroom + one retry, instead of + // 0-retry fail-fast. Starts at 200 GB (fits ~30 nodes) and only escalates to + // 400 GB on an OOM retry (503 GiB group) rather than a flat 400 that would make + // every task compete for the 10 big nodes. Adjust/remove as needed. + withName: '.*similarity.*' { + memory = { [ 200.GB * task.attempt, 400.GB ].min() } + errorStrategy = { task.attempt > 1 ? 'ignore' : 'retry' } + maxRetries = 1 + } +} diff --git a/.claude/skills/run_sweep/SKILL.md b/.claude/skills/run_sweep/SKILL.md new file mode 100644 index 000000000..0e8d6e582 --- /dev/null +++ b/.claude/skills/run_sweep/SKILL.md @@ -0,0 +1,92 @@ +--- +name: run_sweep +description: >- + For a given module (a pipeline stage such as segmentation / transcript_assignment / + cell_type_annotation / expression_correction, a single method, or 'all'), verify every + swept method's build is deploy-fresh (via check-component), then launch each method's + parameter-sweep nebius run, then export a _runs.csv table of (method, + workflow_id, watch_url). Use when the user wants to kick off a stage's parameter sweeps + on the cloud, "start all the runs", (re)launch a method's sweep, or record the + submitted workflow IDs — after tune-method has already produced the param_sweep files. +--- + +# run_sweep + +Launch a module's parameter-sweep runs *safely* — never launch onto a stale container — +and leave behind a table of what was submitted. It composes two things that already exist +instead of reinventing them: + +- **build gate** — [`check-component`](../check-component/SKILL.md)'s `check_component.sh` + (exit 0 = fully deployed on `origin/main` → `build/main` → ghcr `build_main`). +- **launchers** — `scripts/run_benchmark/param_sweep/run_test__nebius.sh` + (produced by [`tune-method`](../tune-method/SKILL.md); each `tw launch`es one method's + star-sweep and prints a `Workflow submitted` line + watch URL). + +This is the automation of what was done by hand for the segmentation stage +(`param_sweep/segmentation_runs.csv`). + +## How to run + +```bash +.claude/skills/run_sweep/run_sweep.sh [--methods m1,m2,...] [--check-only] [--force] [--tag ] +``` + +- `` — a pipeline **stage** (`segmentation`, `transcript_assignment`, + `cell_type_annotation`, `expression_correction`; dashes tolerated), a `methods_*` + namespace, a **single method** name (`stardist`), or `all`. It is also the basename of + the exported CSV → `/_runs.csv` (see Output). +- `--methods` — override the auto-discovered set (e.g. relaunch just the ones that failed). +- `--check-only` — run the build gate and stop; launch nothing. +- `--force` — launch even if a build is stale (default: **abort and launch nothing** if any + build is stale). Use only when you knowingly accept an old container. +- `--tag` — image tag for the build check (default `build_main`; a `build/` deploy + uses `build_`). + +Methods are discovered from the launchers present in `param_sweep/` and filtered to the +requested stage's namespace, so newly-added methods are picked up automatically. + +## What it does, in order + +1. **Resolve** the module to its method set + CSV basename. +2. **[1/3] Build gate** — `check_component.sh ` for each. If any is stale: with + `--check-only` or without `--force`, print which stage failed and **exit without + launching**; with `--force`, warn and continue. +3. **[2/3] Launch** — `bash run_test__nebius.sh` per method, parsing the + `Workflow submitted` line and the `…/watch/` URL from its output. +4. **[3/3] Export** — write the runs table (`method,workflow_id,watch_url`, successful + submissions only) to the external results tree, print it, and report any that failed to + launch (non-zero exit if so). + +## Where results go (convention) + +All sweep-generated files live **outside the git repo**, under +`~/projects/txsim_results/param_sweep_tests//` (override the base with +`$TXSIM_RESULTS_DIR`), one folder per stage: + +- **run tables** (this skill) → `/_runs.csv` +- **parsed run results** (the [`parse-sweep-results`](../parse-sweep-results/SKILL.md) + skill, given a workflow ID) → `/_run__results.csv` + +A run spanning a single stage (a stage keyword, a namespace, or a single method) writes into +that stage's folder; a multi-stage `all`/`--methods` run writes to the tree root. + +## Prerequisites & gotchas + +- **Params must be pushed first.** Each launcher reads its `param_sweep/_params.yaml` + from a **raw-GitHub URL on the current branch** at runtime (see tune-method's params-file + gotcha). The launcher `curl`-checks this and fails fast; if a launch reports a 404, commit + **and push** the params yaml (and confirm the URL path includes `param_sweep/`). +- **`tw` must be authenticated** to the Seqera workspace; launches go to whatever + compute-env/labels each launcher hardcodes (GPU methods keep the `gpu` label + GPU env). +- **Build gate semantics** — the source check compares the component's config on + `origin/main` to your working tree, so an edit that's only on a feature branch (e.g. + `fixes`) reads as *not deployed*: the `build/main` pipeline code the run actually uses + won't contain it. That is correct — merge to `main` and let CI redeploy first. See + check-component's "Interpreting results" for per-stage fixes. +- The script only records **submitted** runs; it does not wait for or judge their outcome. + +## Output + +Report to the user: which methods passed/failed the build gate, the workflow IDs + watch +URLs submitted, the CSV path written, and any launch failures (with the likely cause, e.g. +un-pushed params or auth). Don't reprint the whole script. diff --git a/.claude/skills/run_sweep/run_sweep.sh b/.claude/skills/run_sweep/run_sweep.sh new file mode 100644 index 000000000..e7c458a99 --- /dev/null +++ b/.claude/skills/run_sweep/run_sweep.sh @@ -0,0 +1,162 @@ +#!/usr/bin/env bash +# run_sweep.sh — for a given module (pipeline stage / method set): verify every +# swept method's build is deploy-fresh, then launch its param-sweep nebius run, +# then export a _runs.csv table of (method, workflow_id, watch_url). +# +# It composes two existing pieces rather than reinventing them: +# * the build gate -> .claude/skills/check-component/check_component.sh (exit 0 = OK) +# * the launchers -> scripts/run_benchmark/param_sweep/run_test__nebius.sh +# +# Default behaviour is all-or-nothing: if ANY build is stale, nothing is launched +# (fix the build first, or pass --force). Use --check-only to gate without launching. +# +# Usage: +# run_sweep.sh [--methods m1,m2,...] [--check-only] [--force] [--tag ] +# +# segmentation | transcript_assignment | cell_type_annotation | +# expression_correction (a pipeline stage; dashes ok) +# ... or a methods_* namespace, a single method name, or 'all'. +# Also the basename of the exported CSV (_runs.csv). +# --methods comma/space list overriding the auto-discovered method set. +# --check-only run the build checks and stop (launch nothing). +# --force launch even if some builds are stale (default: abort if any stale). +# --tag image tag passed to check-component (default build_main). +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null)" || { echo "ERROR: not in a git repo"; exit 2; } +cd "$REPO_ROOT" || exit 2 + +SWEEP_DIR="scripts/run_benchmark/param_sweep" +CHECK="$REPO_ROOT/.claude/skills/check-component/check_component.sh" +[ -x "$CHECK" ] || CHECK="$SCRIPT_DIR/../check-component/check_component.sh" + +usage() { sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; } + +MODULE="" ; METHODS_OVERRIDE="" ; CHECK_ONLY=0 ; FORCE=0 ; TAG="build_main" +while [ $# -gt 0 ]; do + case "$1" in + --methods) METHODS_OVERRIDE="${2:-}"; shift 2;; + --check-only) CHECK_ONLY=1; shift;; + --force) FORCE=1; shift;; + --tag) TAG="${2:-}"; shift 2;; + -h|--help) usage; exit 0;; + -*) echo "unknown option: $1"; usage; exit 2;; + *) if [ -z "$MODULE" ]; then MODULE="$1"; else echo "unexpected arg: $1"; exit 2; fi; shift;; + esac +done +[ -z "$MODULE" ] && { usage; exit 2; } +[ -x "$CHECK" ] || { echo "ERROR: check_component.sh not found/executable at: $CHECK"; exit 2; } + +norm="${MODULE//-/_}" + +# method name -> its src namespace (e.g. binning -> methods_segmentation) +method_ns() { + local m="$1" d + d="$(find src -type d -name "$m" 2>/dev/null | grep -v '/target/' | head -1)" + [ -z "$d" ] && return 1 + d="${d#src/}"; echo "${d%/$m}" +} + +# every method that has a param-sweep nebius launcher +all_methods() { + local f b + for f in "$SWEEP_DIR"/run_test_*_nebius.sh; do + [ -e "$f" ] || continue + b="$(basename "$f")"; b="${b#run_test_}"; b="${b%_nebius.sh}" + echo "$b" + done +} + +# ---- resolve method set + CSV basename ---- +METHODS=() ; CSV_BASE="" +if [ -n "$METHODS_OVERRIDE" ]; then + CSV_BASE="${norm#methods_}" + IFS=', ' read -r -a METHODS <<< "$METHODS_OVERRIDE" +elif [ -f "$SWEEP_DIR/run_test_${MODULE}_nebius.sh" ]; then + METHODS=("$MODULE"); CSV_BASE="$MODULE" # single method +elif [ "$norm" = "all" ]; then + CSV_BASE="all"; while read -r m; do METHODS+=("$m"); done < <(all_methods) +else + want_ns="$norm"; [[ "$want_ns" == methods_* ]] || want_ns="methods_$norm" + CSV_BASE="${want_ns#methods_}" + while read -r m; do + ns="$(method_ns "$m" || true)" + [ "$ns" = "$want_ns" ] && METHODS+=("$m") + done < <(all_methods) +fi +[ "${#METHODS[@]}" -eq 0 ] && { echo "ERROR: no sweep launchers matched module '$MODULE'"; exit 2; } + +# ---- output location: external results tree, organised by pipeline stage ---- +# (kept OUT of the git repo; override the base with $TXSIM_RESULTS_DIR) +RESULTS_BASE="${TXSIM_RESULTS_DIR:-$HOME/projects/txsim_results/param_sweep_tests}" +STAGES="$(for m in "${METHODS[@]}"; do ns="$(method_ns "$m" 2>/dev/null || true)"; [ -n "$ns" ] && echo "${ns#methods_}"; done | sort -u)" +n_stages="$(printf '%s\n' "$STAGES" | grep -c .)" +if [ "$n_stages" -eq 1 ]; then RESULTS_DIR="$RESULTS_BASE/$STAGES"; else RESULTS_DIR="$RESULTS_BASE"; fi +mkdir -p "$RESULTS_DIR" +CSV="$RESULTS_DIR/${CSV_BASE}_runs.csv" +echo "Module: $MODULE -> methods: ${METHODS[*]}" +echo "Table: $CSV" +echo + +# ---- [1/3] build gate ---- +echo "===== [1/3] build checks (check-component, tag=$TAG) =====" +stale=() +for m in "${METHODS[@]}"; do + echo "----- $m -----" + if "$CHECK" "$m" "$TAG"; then + echo ">> $m: build OK" + else + rc=$? + echo ">> $m: build NOT fresh (check-component rc=$rc)" + stale+=("$m") + fi + echo +done + +if [ "${#stale[@]}" -gt 0 ]; then + echo "Stale / not-deployed: ${stale[*]}" + [ "$CHECK_ONLY" -eq 1 ] && exit 1 + if [ "$FORCE" -ne 1 ]; then + echo "ABORTING — launching nothing. Fix the build(s) above, or re-run with --force." + exit 1 + fi + echo "WARNING: --force set; launching despite stale build(s)." + echo +fi +if [ "$CHECK_ONLY" -eq 1 ]; then + echo "All builds OK (--check-only). Nothing launched." + exit 0 +fi + +# ---- [2/3] launch (all builds OK here, or --force) ---- +echo "===== [2/3] launching sweeps =====" +rows=() ; failed=() ; launched=0 +for m in "${METHODS[@]}"; do + echo "----- launching $m -----" + out="$(bash "$SWEEP_DIR/run_test_${m}_nebius.sh" 2>&1)"; rc=$? + echo "$out" + wid="$(printf '%s\n' "$out" | grep -oE 'Workflow[[:space:]]+[A-Za-z0-9]+' | awk '{print $2}' | head -1)" + wurl="$(printf '%s\n' "$out" | grep -oE 'https://[^[:space:]]+/watch/[A-Za-z0-9]+' | head -1)" + if [ -n "$wid" ]; then + rows+=("$m,$wid,$wurl"); launched=$((launched+1)) + echo ">> $m submitted: $wid" + else + failed+=("$m") + echo ">> $m LAUNCH FAILED (rc=$rc, no workflow id in output)" + fi + echo +done + +# ---- [3/3] export table ---- +echo "===== [3/3] exporting table =====" +{ + echo "method,workflow_id,watch_url" + for r in ${rows[@]+"${rows[@]}"}; do echo "$r"; done +} > "$CSV" +echo "Wrote $CSV" +cat "$CSV" +echo +echo "Launched $launched / ${#METHODS[@]}" +[ "${#failed[@]}" -gt 0 ] && { echo "FAILED to launch: ${failed[*]}"; exit 1; } +exit 0 diff --git a/.claude/skills/tune-method/SKILL.md b/.claude/skills/tune-method/SKILL.md new file mode 100644 index 000000000..78aa7654c --- /dev/null +++ b/.claude/skills/tune-method/SKILL.md @@ -0,0 +1,187 @@ +--- +name: tune-method +description: >- + Study a benchmark method component in depth, identify the parameters that matter for + optimization (with explicit quality-vs-speed criteria), and set up a parameter-sweep test + run for it. Three phases: (1) collect knowledge — git history, config-linked docs, upstream + repo/tutorial, and the ACTUAL publication; (2) identify optimization parameters + ranges + with criteria, exposing high-value knobs the component doesn't yet surface; (3) generate a + cloud (nebius) run script + committed params yaml under scripts/run_benchmark/param_sweep/. + Use when the user wants to + optimize/tune a method, run a parameter sweep, "study everything on ", find which + knobs matter, or prep a benchmark run that varies a method's parameters. +--- + +# tune-method + +Turning a benchmark method into a *tunable* one is three phases that build on each other. +Do them in order — the parameter choices in phase 2 must be grounded in phase 1's evidence, +and the sweep in phase 3 is only as good as phase 2's criteria. + +| Phase | Goal | Output | +|-------|------|--------| +| 1. Collect knowledge | understand the method, its defaults, and the real upstream behaviour | an accurate `NOTES.md` (+ citation caveats) | +| 2. Identify parameters + criteria | decide which knobs matter and over what range, expose missing ones | an "Optimization / tuning" section; possibly new config args | +| 3. Set up the sweep | make it runnable | a cloud `run_test__nebius.sh` + committed `_params.yaml`, both under `scripts/run_benchmark/param_sweep/` | + +Scale the effort to the ask: a quick "what are the knobs" stops after phase 2; "set up a +sweep to optimize X" runs all three. + +--- + +## Phase 1 — Collect knowledge + +Resolve the method to `src///` (e.g. `methods_segmentation/cellposev4`). +Then read, in this order, and **don't skip the paper**: + +1. **`config.vsh.yaml`** — the exposed `arguments:` and their defaults, the `engines[docker]` + setup (version pins — often load-bearing), `links.documentation` / `links.repository`, + and `references.doi`. +2. **`script.py` / `script.R`** — which args are *actually* forwarded to the underlying + tool vs hardcoded/ignored, and what the input contract is (e.g. only `image[0]` — a single + channel — is segmented). The gap between "config arg" and "what the tool call receives" is + where real behaviour lives. +3. **Git history** — `git log --oneline -- ` and read the introducing commit(s). This is + how you learn *why* defaults are the way they are — frequently they are **deliberately + speed-tuned** to fit a CI/GPU time budget (that context is the whole basis for phase 2). +4. **The config's linked docs** — fetch `links.documentation` (settings / API pages give the + real parameter list, defaults, and quality-vs-speed notes) and `links.repository`. +5. **The actual publication — verify, do not trust `references.doi` blindly.** The DOI in the + config is often the *generic/framework* paper, not the one describing the model actually + run (real example: `cellposev4`'s DOI pointed at the original 2021 Cellpose paper, not the + Cellpose-SAM preprint it runs). Find and READ the method-specific paper; cross-check claims + against its abstract rather than paraphrasing a README or search snippet. + - bioRxiv/medRxiv **block WebFetch (403)** → use the bioRxiv MCP `get_preprint` tool by DOI + for title/authors/abstract/metadata. For journals, WebFetch the DOI (follow the redirect + to the publisher host on the second call) or use the PubMed MCP. + - If the config DOI is wrong or generic, **flag it and offer to fix** — cite both the + method-specific paper *and* the framework paper (Viash `references.doi` accepts a list). + +**Write it down in `NOTES.md`.** Use the **`document-method-troubleshooting`** skill for the +NOTES.md + memory mechanics and section conventions — do not reinvent them here. Add a +**"Citation caveat"** near the top if the DOI didn't match, and record which defaults are +speed-tuned (you'll need that in phase 2). + +## Phase 2 — Identify optimization parameters + criteria + +Enumerate **every** knob the underlying tool's `eval`/API exposes (from phase 1's docs), then +organise them. The organising principle is *impact tiers* — this IS the criteria: + +- **Tier 0 — the input, not a parameter.** Often the biggest lever: channels fed, resolution, + which stain. (e.g. feeding a 2-channel membrane+nuclear stack vs a single channel.) Call it + out even though it isn't in the sweep. +- **Tier 1 — highest impact on output quality.** The knobs that change *what* gets detected + (recall↔precision dials, size/scale params). +- **Tier 2 — quality/speed trade-offs.** Knobs that mostly buy accuracy at a time cost. +- **Tier 3 — not exposed by the component.** High-value knobs worth *adding* to the config + for a serious sweep. + +Three rules that come straight from phase 1: + +- **Defaults are usually speed-tuned**, so "optimize for quality" mostly means *walking them + back toward the tool's own defaults* (re-enable a QC check that was disabled for speed, raise + an iteration count that was floored, etc.). Say so. +- **Ground every default and range in evidence** (docs + paper), not memory. Tie ranges to the + data when you can (e.g. pick a `diameter` range from the pixel size: Xenium ~0.2125 µm/px ⇒ + nuclei ~35–40 px, whole cells ~60–70 px). +- **Any parameter already set to a non-default value is a sweep candidate, not a silent + baseline.** Compare each config default (and anything the run script's `default:` block sets) + against the *tool's* true default. A deviation is evidence someone already found the knob + matters — so it must go on a sweep axis, with its range straddling both the deviated value + and the tool default. The **only** exception is strictly-performance / resource knobs that + can't change the output (e.g. cores/threads, `batch_size`, `n_workers`, memory, tile size for + pure memory management) — those stay fixed in `default:` and never enter the sweep. + +**Criteria for a good sweep axis** (used in phase 3): + +- Continuous / multi-level params get a **range of several values**. +- **Boolean** params have exactly **one** meaningful non-default value (the flip) — don't pad + them. +- **Omit the default value from the sweep list** — the "default variant" already covers that + point (see phase 3); repeating it just runs a duplicate. + +**If a Tier-1/Tier-3 knob isn't exposed, expose it.** Add it to `config.vsh.yaml`'s +`arguments:` (type, default = the tool's default, a clear description) and thread it into the +script's eval-params (match the existing pattern — e.g. add the key to the params-forwarding +comprehension). Validate with `viash config view `. Record the new arg and the tier +ranking in NOTES.md under an **"Optimization / tuning"** section. + +## Phase 3 — Set up the sweep run scripts + +### How the sweep expands (know this cold) + +The `run_benchmark` workflow reads a `method_parameters_yaml` file +(`src/workflows/run_benchmark/main.nf`, init map ~L22-32) shaped as: + +```yaml +parameters: + : + default: { argA: ..., argB: ... } # baseline args applied to EVERY variant + sweep: { argA: [v1, v2, v3] } # one extra variant per value, that ONE arg overridden +``` + +The expansion (main.nf ~L738-773) is a **star around the default, NOT a grid**: sweep list +lengths **add**, they do not multiply. **Total variants = 1 (default) + Σ(len of each sweep +list).** There's also a hard constraint — **at most one non-default method/variant at a time** +across the whole pipeline — so the swept method must be the only non-default thing; keep every +other stage on its single default. + +### Build the script + +Cloud/nebius only — **do NOT generate a local `run_test__local.sh`**. Everything the +sweep needs lives under **`scripts/run_benchmark/param_sweep/`** (create the folder if missing): + +- **`param_sweep/_params.yaml`** — the `method_params` (default + sweep) as a committed + file, NOT an inline heredoc. This is the single source of truth for the sweep. +- **`param_sweep/run_test__nebius.sh`** — copy `run_test_nebius.sh` / `run_gpu_nebius.sh` + (`tw launch`); GPU methods also keep the `gpu` label + GPU compute env. + +In the settings block: enable **all default methods + the target method** at its stage +(comment the rest), and set `method_parameters_yaml` to the committed params file's raw-GitHub +URL (see below). + +### The params-file placement gotcha (cloud) + +`readYaml` uses Nextflow's `file()` (`common/nextflow_helpers/workflowHelper.nf`), and the +workflow opens `method_parameters_yaml` **at runtime on the cloud compute env**, so a local +path (e.g. `/tmp`) does not exist there — this is why the binning `method_params` block is +commented out in the stock `run_test_nebius.sh`. Note `/scratch` is **read-only from the launch +host** (the cloud writes results there, you can't stage into it), so it is NOT a staging +option. `file()` does stage **`http(s)://`** and **`s3://`**, so use one of: + +- **Committed file + raw GitHub URL (preferred).** `tw launch` already pulls the repo from + GitHub, so commit `scripts/run_benchmark/param_sweep/_params.yaml` and set + `method_parameters_yaml` to its + `https://raw.githubusercontent.com////scripts/run_benchmark/param_sweep/_params.yaml` + URL (public repo ⇒ no auth). The file must be **pushed to `` before launching**; + `curl -fsSL` it first in the script to fail fast if it's not there yet. This is independent + of `--revision` (which selects the pipeline *code*). +- **`s3://` URI** — upload the params to a bucket the launcher can write and the compute env + can read (`aws s3 cp`, possibly `--profile`). + +GPU methods also need the `gpu` label + GPU compute env. + +### Validate before handing back + +- `bash -n param_sweep/run_test__nebius.sh` (syntax). +- **Non-default audit (do this before finalizing the sweep).** Walk every argument in + `config.vsh.yaml` and every key in the params `default:` block, and compare its effective + value to the *tool's* true default (from phase 1). For each one that deviates: + - if it's a strictly-performance / resource knob (cores, threads, `batch_size`, `n_workers`, + memory, memory-only tiling) → leave it fixed and note it; + - otherwise → it **must** appear as a `sweep:` axis. If it doesn't, add it (range straddling + the deviated value and the tool default, per the phase-2 rule) before handing back. + List what you found and how you resolved it. +- Run `validate_sweep.py` (in this skill dir) on `param_sweep/_params.yaml` to print the + per-param value counts, the **total variant count**, and a warning if any sweep list repeats + its default or if the sweep is heavy for a "test" run. +- `curl -fsSL` the raw-GitHub params URL to confirm it resolves once the file is pushed. +- Sanity-check the variant count against the run's budget — a 19-variant sweep × the full + downstream pipeline is a lot for a quick smoke test; note it and offer to trim. + +## Output + +Tell the user, briefly: the NOTES.md sections written, any config args newly exposed (and that +they need `viash ns build` + a container rebuild to take effect — see `check-component`), the +`param_sweep/` files created (params yaml + nebius script), and the resulting variant count. +Don't reprint whole files. diff --git a/.claude/skills/tune-method/validate_sweep.py b/.claude/skills/tune-method/validate_sweep.py new file mode 100644 index 000000000..8c7fd16df --- /dev/null +++ b/.claude/skills/tune-method/validate_sweep.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 +"""Validate a run_benchmark `method_params` sweep block. + +Usage: + validate_sweep.py [component_name] + +The file is the one referenced by `method_parameters_yaml` — a mapping shaped as: + + parameters: + : + default: { arg: val, ... } + sweep: { arg: [v1, v2, ...], ... } + +For each method it reports the default args, each swept parameter's value count, and the +TOTAL number of variants the workflow will expand — which is `1 (default) + Σ len(sweep list)` +because the benchmark varies ONE parameter at a time (a "star" around the default, not a +grid). It warns when a sweep list repeats the default value (a redundant duplicate variant) +and when the sweep is heavy for a quick test run. + +Exit status is non-zero if the file has no usable `parameters:` block, so it can gate a script. +""" +import sys +import yaml + + +def check_method(comp, spec): + spec = spec or {} + default = spec.get("default", {}) or {} + sweep = spec.get("sweep", {}) or {} + print(f"\n== {comp} ==") + print(f" default: {default or '(none)'}") + n_variants = 1 # the default variant + for par, vals in sweep.items(): + if not isinstance(vals, list): + vals = [vals] + n_variants += len(vals) + # bool must be checked before the general == (True == 1 in Python) + dups = [ + v for v in vals + if par in default + and type(v) is type(default[par]) + and v == default[par] + ] + warn = "" + if dups: + warn = f" ** repeats default ({default[par]}) -> redundant variant, drop it" + elif isinstance(default.get(par), bool): + warn = " (boolean: one non-default value is all there is)" + print(f" sweep {par}: {len(vals)} value(s) {vals}{warn}") + print(f" -> variants: 1 default + {n_variants - 1} sweep = {n_variants}") + return n_variants + + +def main(argv): + if len(argv) < 2: + print("usage: validate_sweep.py [component]", file=sys.stderr) + return 2 + with open(argv[1]) as fh: + doc = yaml.safe_load(fh) + only = argv[2] if len(argv) > 2 else None + + params = doc.get("parameters", doc) if isinstance(doc, dict) else None + if not isinstance(params, dict) or not params: + print("no usable `parameters:` block found", file=sys.stderr) + return 1 + + total = 0 + seen = False + for comp, spec in params.items(): + if only and comp != only: + continue + seen = True + total += check_method(comp, spec) + if only and not seen: + print(f"component '{only}' not found in file", file=sys.stderr) + return 1 + + print(f"\nTOTAL variants across method(s): {total}") + if total > 12: + print("NOTE: >12 variants -> heavy for a quick test run; consider trimming lists.") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..749942bd3 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,117 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +This is an **OpenProblems v2 benchmarking task** for preprocessing imaging-based spatial transcriptomics (iST) data. It uses **Viash** (containerized component framework) and **Nextflow** (workflow orchestration) to benchmark methods across a multi-stage preprocessing pipeline: segmentation → transcript assignment → count aggregation → QC filtering → cell volume calculation → normalization → cell type annotation → expression correction. + +## Key Commands + +### Build & Test + +```bash +# Build all components (creates executables in target/) +viash ns build --parallel + +# Build Docker containers (required before running benchmarks) +scripts/project/build_all_docker_containers.sh +# Equivalent: viash ns build --parallel --setup cachedbuild + +# Test a single component +viash test src/methods_segmentation/binning/config.vsh.yaml + +# Test all components in parallel +viash ns test --parallel +# or via script: +scripts/project/test_all_components.sh + +# Filter tests by namespace +viash ns test --parallel -q methods_segmentation +``` + +### Run Benchmarks + +```bash +# Local test run (uses test resources, outputs to temp/results/) +scripts/run_benchmark/run_test_local.sh + +# Full local benchmark (uses resources/, outputs to resources/results/) +scripts/run_benchmark/run_full_local.sh + +# Cloud runs (Nebius, Seqera) +scripts/run_benchmark/run_test_nebius.sh +scripts/run_benchmark/run_test_seqeracloud.sh +``` + +### Create Test Resources + +```bash +# Download and process test datasets (run via Nextflow) +scripts/create_test_resources/.sh +``` + +## Architecture + +### Component Structure + +Every method/metric is a self-contained Viash component: + +``` +src/// + config.vsh.yaml # Component spec: arguments, Docker image, resources + script.py # Implementation (or script.R) +``` + +`config.vsh.yaml` merges the API interface via `__merge__: /src/api/comp_.yaml`, which enforces standard input/output arguments. Component-specific parameters go in the `arguments:` block. + +### API Layer (`src/api/`) + +- `comp_method_*.yaml` — Defines the standard interface (input/output args) for each pipeline stage +- `file_*.yaml` — Defines file format specifications +- Components inherit these via `__merge__` in their config + +### Data Formats + +All spatial data flows as `.zarr` (SpatialData format); single-cell references and outputs use `.h5ad` (AnnData). Key file types: +- `raw_ist.zarr` → raw imaging data +- `segmentation.zarr` → cell label maps +- `transcript_assignments.zarr` → per-transcript cell assignments +- `spatial_aggregated_counts.h5ad` → cell × gene matrix with centroids/volume +- `scrnaseq_reference.h5ad` → reference atlas for annotation + +### Pipeline Flow + +The benchmark workflow (`src/workflows/run_benchmark/main.nf`) fans out across all enabled methods at each stage, running every combination. The dataset processor (`src/workflows/process_datasets/main.nf`) converts raw vendor outputs to the standardized `raw_ist.zarr` format. + +### Key Dependencies + +- `txsim` (from `theislab/txsim@dev`) — core spatial transcriptomics utilities used in most scripts +- `spatialdata>=0.7.3` — standardized spatial data container +- `anndata>=0.12.0`, `zarr>=3.0.0` + +Base Docker setups are mixed in via `__merge__` in component configs: +- `src/base/setup_spatialdata_partial.yaml` — spatialdata + dependencies +- `src/base/setup_txsim_partial.yaml` — txsim library + +### Resource Labels (Nextflow) + +Components are tagged with resource labels in their `runners[nextflow].directives`: +- Memory: `lowmem` (20 GB), `midmem` (50 GB), `highmem` (100 GB) +- CPU: `lowcpu` (5), `midcpu` (15), `highcpu` (30) +- Time: `lowtime` (1h), `midtime` (4h), `hightime` (8h), `veryhightime` (24h) + +### Test Resources + +Located in `resources_test/` — downloaded from S3 (`s3://openproblems-data/resources_test/`). The `_viash.yaml` defines the S3 paths; use `viash run` with the appropriate config to sync them. + +### Common Submodule + +`common/` is a git submodule from `openproblems-bio/openproblems`. It contains shared component test scripts (`common/component_tests/`) that validate output format compliance. + +## Adding a New Method + +1. Create `src///config.vsh.yaml` — merge the appropriate `comp_method_*.yaml` API +2. Create `script.py` implementing the logic; read inputs via `par["input_*"]`, write outputs via `par["output_*"]` +3. Add `test_resources` pointing to `resources_test/` data and a `common/component_tests/` test runner +4. Register the method in `scripts/run_benchmark/config.yaml` and the run scripts diff --git a/PIPELINE_OVERVIEW.md b/PIPELINE_OVERVIEW.md new file mode 100644 index 000000000..ae2dfaca7 --- /dev/null +++ b/PIPELINE_OVERVIEW.md @@ -0,0 +1,236 @@ +# iST Preprocessing Benchmark — Pipeline Elements + +> Reference sheet describing every element of the `task_ist_preprocessing` benchmark, for use as an input to visualization/design work. Organized so each block (nodes, stages, methods, formats, skills) can be turned directly into diagram elements. + +--- + +## 0. What this is + +An **OpenProblems v2 benchmarking task** for preprocessing **imaging-based spatial transcriptomics (iST)** data (10x Xenium, Vizgen MERSCOPE, Bruker CosMx, Allen MERFISH). + +**Tech stack (the "rails" everything runs on):** + +| Layer | Tool | Role | +|-------|------|------| +| Component framework | **Viash** | Each method/metric is a containerized component (`config.vsh.yaml` + `script.py`/`.R`) | +| Orchestration | **Nextflow** | Fans components out across the benchmark matrix | +| Containers | **Docker → ghcr.io** | One image per component, tagged `build_main` | +| Data containers | **SpatialData `.zarr`** (spatial) + **AnnData `.h5ad`** (single-cell / tabular) | +| Core library | **txsim** (`theislab/txsim@dev`) | Shared spatial-transcriptomics utilities | + +There are **two pipelines**: (A) dataset creation and (B) the benchmark itself. + +--- + +## 1. Pipeline A — Dataset Creation (`process_datasets`) + +Two parallel input paths (spatial + single-cell) that meet at a **combine** step. Produces the standardized inputs the benchmark consumes. + +``` + SPATIAL PATH (iST) SINGLE-CELL PATH (scRNA-seq reference) + ───────────────── ───────────────────────────────────── + vendor raw data vendor raw data + │ [spatial loader] │ [SC loader] (counts only) + ▼ ▼ + dataset.zarr (raw SpatialData) dataset.h5ad (counts) + │ [crop_region] (optional) │ log_cp → hvg → pca → knn + ▼ ▼ + cropped dataset.zarr processed reference.h5ad (counts + normalized + HVG/PCA/kNN) + │ │ + └───────────────┬──────────────────────────┘ + ▼ + [ process_dataset ] ← intersect shared genes, crop huge images, rechunk + ▼ + raw_ist.zarr + scrnaseq_reference.h5ad ← inputs to Pipeline B +``` + +### Loaders (`src/datasets/loaders/`) + +**Spatial (iST) loaders** — raw vendor output → raw SpatialData `dataset.zarr`: +- `tenx_xenium` — 10x Xenium +- `tenx_atera` — 10x Xenium (Atera / newer format) +- `vizgen_merscope` — Vizgen MERSCOPE +- `bruker_cosmx` — Bruker CosMx +- `bruker_cosmx_nsclc` — Bruker CosMx (NSCLC) +- `allen_brain_cell_atlas_merfish` — Allen MERFISH + +**Single-cell (reference) loaders** — raw → `.h5ad` (**counts only**; normalization added later): +- `allen_brain_cell_atlas` +- `andrews_human_liver_sc` +- `ganier_human_skin_sc` +- `lee_human_colon_cancer_sc` +- `lu_human_liver_cancer_sc` +- `travaglini_human_lung_sc` +- `wu_human_breast_cancer_sc` +- `zuani_human_nsclc_sc` + +### Processors & shared components +- `crop_region` — optional spatial crop (local processor) +- Shared (pulled from external repos): `normalization/log_cp`, `processors/hvg | pca | knn`, `utils/extract_uns_metadata` + +### Per-dataset workflows (`src/datasets/workflows/process_`) +- **SC workflow:** loader → `log_cp` (adds log-CP10k `normalized` layer) → `hvg` → `pca` → `knn` → `extract_uns_metadata`. Also a generic loader-agnostic `process_scrnaseq`. +- **Spatial workflow:** loader → optional `crop_region` → `dataset.zarr` + +### Combine (`process_dataset`) +Intersects spatial + SC on shared genes (`feature_name`), crops oversized images (> ~20000² px), rechunks to a uniform grid, renames `table`→`metadata`. **Does not normalize.** + +--- + +## 2. Pipeline B — The Benchmark (`run_benchmark`) + +The core preprocessing pipeline. **Data format changes partway through:** SpatialData `.zarr` through transcript assignment, then AnnData `.h5ad` from count aggregation onward. + +### Core stage flow + +``` + raw_ist.zarr + │ + ① SEGMENTATION ──────────────────► segmentation.zarr + │ + ② TRANSCRIPT ASSIGNMENT ─────────► transcript_assignments.zarr (+ raw_ist.zarr) + │ ⋮ format switch .zarr → .h5ad + ③ COUNT AGGREGATION ─────────────► spatial_aggregated_counts.h5ad + │ + ├─④ QC FILTER ───────────────────► spatial_qc_col.h5ad + │ + ├─⑤ CELL VOLUME ─────────────────► cell_volumes.h5ad + │ + ⑥ NORMALIZATION ─────────────────► spatial_normalized_counts.h5ad + │ + ⑦ CELL TYPE ANNOTATION ──────────► spatial_with_cell_types.h5ad (+ scrnaseq_reference.h5ad) + │ + ⑧ EXPRESSION CORRECTION ─────────► spatial_corrected_counts.h5ad + │ + ⑨ GENE EFFICIENCY CORRECTION ────► (corrected counts) + │ + ⑩ DATA AGGREGATION ──────────────► spatial_processed_complete.zarr + │ + ⑪ METRICS ──────────────────────► score.h5ad +``` + +### Stage-by-stage detail + +| # | Stage (namespace) | Purpose | Methods available | +|---|-------------------|---------|-------------------| +| ① | **Segmentation** (`methods_segmentation`) | Partition the image into cell label maps | `binning`, `cellpose`, `cellposev4`, `custom_segmentation`, `stardist`, `watershed` | +| ② | **Transcript assignment** (`methods_transcript_assignment`) | Assign each transcript to a cell | `basic_transcript_assignment`, `baysor`, `clustermap`, `comseg`, `fastreseg`, `pciseq`, `proseg`, `segger` | +| ③ | **Count aggregation** (`methods_count_aggregation`) | Aggregate transcripts → cell × gene counts | `basic_count_aggregation` | +| ④ | **QC filter** (`methods_qc_filter`) | Flag cells passing quality control | `basic_qc_filter` | +| ⑤ | **Cell volume** (`methods_calculate_cell_volume`) | Compute per-cell volume | `alpha_shapes` | +| ⑥ | **Normalization** (`methods_normalization`) | Normalize expression | `normalize_by_counts`, `normalize_by_volume`, `spanorm` | +| ⑦ | **Cell type annotation** (`methods_cell_type_annotation`) | Label cell types (uses scRNA-seq reference) | `mapmycells`, `moscot`, `rctd`, `singler`, `ssam`, `tacco`, `tangram` | +| ⑧ | **Expression correction** (`methods_expression_correction`) | Correct/denoise expression | `no_correction`, `resolvi_correction`, `split` | +| ⑨ | **Gene efficiency correction** (`methods_gene_efficiency_correction`) | Correct per-gene detection efficiency | `gene_efficiency_correction`, `no_correction` | +| ⑩ | **Data aggregation** (`methods_data_aggregation`) | Merge raw + processed files into one zarr | `aggregate_spatial_data` | + +> Method counts: **6 segmentation · 8 transcript assignment · 7 cell-type annotation** are the "wide" stages; most others have 1–3. + +--- + +## 3. Control Methods (`control_methods`) + +Baselines that **bypass the entire pipeline** — built directly from the scRNA-seq reference, mixed in just before scoring: +- `identity` — identical copy of the scRNAseq reference (positive control / upper bound) +- `permute_celltype_annotations` — randomly permuted cell-type labels (negative control / lower bound) + +Controls only produce the files the **similarity** metric needs, so **controls receive only similarity scores, not quality scores.** + +--- + +## 4. Metrics (`metrics`) + +Two metric components, each emitting several sub-metrics into `score.h5ad`. + +### `quality` — reference-free (gated on data aggregation) +- `proportion_of_assigned_reads` — proportion of assigned reads +- `proportion_of_annotated_cells` — proportion of cells annotated with a cell type +- `number_of_cells` — count of cells in the spatial dataset + +### `similarity` — reference-based (spatial vs. scRNA-seq reference) +- `negative_marker_purity_reads` — % negative-marker reads assigned to correct cell types +- `negative_marker_purity_cells` — % cells without negative-marker counts for their type +- `coexpr_similarity` — similarity of co-expression patterns (spatial vs. sc) +- `coexpr_similarity_celltype` — within-cell-type co-expression similarity +- `rel_pairwise_ct_expr_sim` — mean expression difference between cell-type pairs +- `rel_pairwise_gene_expr_sim` — mean expression difference between gene pairs +- `knn_mixing` — modality mixing in the joint kNN graph + +--- + +## 5. Data Formats (nodes between stages) + +| File | Container | Produced by | Contents | +|------|-----------|-------------|----------| +| `raw_ist.zarr` | SpatialData | Pipeline A | Raw imaging + transcripts (benchmark entry point) | +| `scrnaseq_reference.h5ad` | AnnData | Pipeline A | SC reference: counts + normalized + HVG/PCA/kNN | +| `segmentation.zarr` | SpatialData | ① | Cell label maps | +| `transcript_assignments.zarr` | SpatialData | ② | Per-transcript cell assignment (point cloud) | +| `spatial_aggregated_counts.h5ad` | AnnData | ③ | Raw cell × gene counts | +| `spatial_qc_col.h5ad` | AnnData | ④ | QC pass/fail column | +| `cell_volumes.h5ad` | AnnData | ⑤ | Per-cell volume column | +| `spatial_normalized_counts.h5ad` | AnnData | ⑥ | Normalized expression layer | +| `spatial_with_cell_types.h5ad` | AnnData | ⑦ | Counts + normalized + cell types | +| `spatial_corrected_counts.h5ad` | AnnData | ⑧ | Corrected + uncorrected normalized expression | +| `spatial_processed_complete.zarr` | SpatialData | ⑩ | All raw + processed data combined | +| `score.h5ad` | AnnData | ⑪ | Metric ids + values | + +--- + +## 6. Orchestration Invariants (how the matrix runs) + +Important for accurately depicting the DAG: + +- **Star-matrix, NOT a full cartesian product.** At most one non-default method per run — the matrix varies **one stage at a time** around a default pipeline. +- **Controls bypass the pipeline** and are injected just before scoring. +- **Quality metric is gated** on the data-aggregation output; `aggregate_spatial_data` is its sole feeder. +- **Two normalization branches:** the standard volume path, plus an active "direct-normalization" branch (`normalize_by_counts` + `spanorm`) that skips cell-volume. +- **Direct-assignment branch is inactive** (empty / commented out). +- **Scores output:** `score_uns.yaml` (+ `dataset_uns`, `method_configs`, `metric_configs`, `task_info`, `trace.txt`) per run. + +### Nextflow resource labels (for annotating compute cost) +- **Memory:** `lowmem` 20 GB · `midmem` 50 GB · `highmem` 100 GB +- **CPU:** `lowcpu` 5 · `midcpu` 15 · `highcpu` 30 +- **Time:** `lowtime` 1h · `midtime` 4h · `hightime` 8h · `veryhightime` 24h +- Some methods (e.g. `segger`) are **GPU-only**. + +--- + +## 7. Deployment Model (three gates before a run) + +A component change only reaches a cluster run after **three** independent things happen — a failure at any gate means the cluster keeps using the **old container**: + +``` + [1] SOURCE → [2] CODEGEN → [3] CONTAINER + committed to build/main deploy Docker image pushed to + origin/main branch regenerated ghcr.io @ build_main tag + (inlined main.nf) (revision label = git SHA) +``` + +The decisive freshness signal is the image's `org.opencontainers.image.revision` label vs. `origin/main` HEAD. + +--- + +## 8. Claude Code Skills (repo tooling, `.claude/skills/`) + +Custom skills that support the *development/ops* workflow around the pipeline (not part of the data flow itself): + +| Skill | What it does | When it fires | +|-------|--------------|---------------| +| **check-component** | Verifies a component is fully deployed across the 3 gates (source → codegen → container); reads the ghcr revision label without pulling the image | Before (re)running a benchmark; when an edit seems to have no effect; to confirm an image isn't stale | +| **debug-component-k8s** | Runs a component's container **live on the k8s cluster** to reproduce and fix runtime environment errors (ImportError, missing `.so`, CLI API changes, base-image issues) | When a component fails *inside* its image at runtime and needs reproduction without a full Nextflow run | +| **document-method-troubleshooting** | Captures troubleshooting into a two-layer record: in-repo `NOTES.md` (authoritative how/why/where-it-breaks) + a memory file (pointer + dated iteration log) | After debugging a component; "document this / write it up / add to the notes"; append after each further iteration | + +--- + +## Quick element inventory (for legends / counts) + +- **Pipeline stages (core):** 10 (① segmentation → ⑩ data aggregation) + metrics +- **Total method components:** ~30 across all stages +- **Widest stages:** transcript assignment (8), cell-type annotation (7), segmentation (6) +- **Control methods:** 2 +- **Metric components:** 2 (quality = 3 sub-metrics, similarity = 7 sub-metrics) +- **Spatial vendors:** 4 (Xenium, MERSCOPE, CosMx, MERFISH) +- **Dataset loaders:** 6 spatial + 8 single-cell +- **Data formats:** 2 containers (`.zarr` SpatialData, `.h5ad` AnnData) across 12 file specs +- **Dev skills:** 3 diff --git a/scripts/create_resources/combine/process_datasets_kuppe_kidney_nebius.sh b/scripts/create_resources/combine/process_datasets_kuppe_kidney_nebius.sh new file mode 100644 index 000000000..f745609af --- /dev/null +++ b/scripts/create_resources/combine/process_datasets_kuppe_kidney_nebius.sh @@ -0,0 +1,78 @@ +#!/bin/bash + +# Process ONLY the Kuppe kidney MERFISH datasets (combine step) — TWO condition-specific pairings. +# Unlike the LTX/MPII combine scripts (one shared SC reference across samples), each Kuppe sample +# is paired with its OWN condition-matched SC reference, because the cancer and healthy releases use +# disjoint cell-type vocabularies AND different gene panels: +# cancer : MERFISH 500-gene Pan-Cancer panel <-> kuppe_kidney_cancer_sc (Flex snRNA-seq, Cancer) +# healthy : MERFISH 140-gene Spapros panel <-> kuppe_kidney_healthy_sc (Flex snRNA-seq, Healthy) +# +# Reads each spatial input (process_vizgen_merscope loader output) from the local /scratch raw +# folder produced by process_kuppe_kidney_merfish_nebius.sh, and combines it with the matching +# processed SC reference produced by process_kuppe_kidney_sc_nebius.sh. +# +# Prerequisites (both publish to the same /scratch raw folder used below): +# 1. scripts/create_resources/spatial/process_kuppe_kidney_merfish_nebius.sh +# -> /scratch/.../raw/vizgen_merscope/kuppe_kidney_{cancer,healthy}_merfish/rep1/dataset.zarr +# 2. scripts/create_resources/sc/process_kuppe_kidney_sc_nebius.sh (log_cp -> hvg -> pca -> knn, +# i.e. it must carry a 'normalized' layer) +# -> /scratch/.../raw/kuppe_kidney_{cancer,healthy}_sc/annotated/dataset.h5ad + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# process_kuppe_kidney_merfish_nebius.sh + process_kuppe_kidney_sc_nebius.sh both publish here. +raw_dir='/scratch/task_ist_preprocessing/raw' +publish_dir='/scratch/task_ist_preprocessing/datasets' + +launch_batch() { + local params_file="$1" + local label="$2" + tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/process_datasets/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file "$params_file" \ + --config src/base/labels_nebius.config \ + --labels "task_ist_preprocessing,process_datasets,$label" +} + +cat > /tmp/params_kuppe_kidney.yaml << HERE +param_list: + + - id: "kuppe_kidney_cancer_merfish_combined/rep1" + input_sp: "$raw_dir/vizgen_merscope/kuppe_kidney_cancer_merfish/rep1/dataset.zarr" + input_sc: "$raw_dir/kuppe_kidney_cancer_sc/annotated/dataset.h5ad" + dataset_id: "kuppe_kidney_cancer_merfish_combined/rep1" + dataset_name: "Kidney cancer combined Kuppe MERFISH + Kuppe snRNAseq (Cancer)" + dataset_url: "" + dataset_reference: "" + dataset_summary: "Kuppe Clear Cell Kidney Carcinoma MERFISH (Vizgen MERSCOPE, 500-gene Pan-Cancer panel) + Kuppe kidney cancer snRNAseq reference" + dataset_description: "Kuppe lab Clear Cell Kidney Carcinoma FFPE sample (MERFISH, Vizgen MERSCOPE, 500-gene Pan-Cancer panel) paired with the Kuppe kidney cancer Flex snRNA-seq reference. Note: the Pan-Cancer panel does not separate NKT vs T cells or pericytes vs vSMC." + dataset_organism: "homo_sapiens" + + - id: "kuppe_kidney_healthy_merfish_combined/rep1" + input_sp: "$raw_dir/vizgen_merscope/kuppe_kidney_healthy_merfish/rep1/dataset.zarr" + input_sc: "$raw_dir/kuppe_kidney_healthy_sc/annotated/dataset.h5ad" + dataset_id: "kuppe_kidney_healthy_merfish_combined/rep1" + dataset_name: "Kidney healthy combined Kuppe MERFISH + Kuppe snRNAseq (Healthy)" + dataset_url: "" + dataset_reference: "" + dataset_summary: "Kuppe healthy kidney MERFISH (Vizgen MERSCOPE, 140-gene Spapros panel) + Kuppe kidney healthy snRNAseq reference" + dataset_description: "Kuppe lab healthy kidney FFPE sample (MERFISH, Vizgen MERSCOPE, 140-gene Spapros panel) paired with the Kuppe kidney healthy Flex snRNA-seq reference." + dataset_organism: "homo_sapiens" + +output_sc: "\$id/output_sc.h5ad" +output_sp: "\$id/output_sp.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +launch_batch /tmp/params_kuppe_kidney.yaml "kuppe_kidney" diff --git a/scripts/create_resources/combine/process_datasets_ltx_nebius.sh b/scripts/create_resources/combine/process_datasets_ltx_nebius.sh new file mode 100644 index 000000000..2966f0486 --- /dev/null +++ b/scripts/create_resources/combine/process_datasets_ltx_nebius.sh @@ -0,0 +1,72 @@ +#!/bin/bash + +# Process ONLY the custom LTX human-lung Xenium dataset (combine step) — TWO samples. +# Reads each spatial input (process_tenx_xenium loader output) from the local +# /scratch raw folder produced by process_ltx_xenium_nebius.sh, and combines each +# with the (single, shared) LTX human lung scRNAseq reference. +# +# The scRNAseq reference must already be standardized (scripts/create_resources/ +# sc/standardize_ltx_sc.py) AND processed (log_cp -> hvg -> pca -> knn, i.e. it +# must carry a 'normalized' layer) before it is used here — run +# scripts/create_resources/sc/process_ltx_sc_nebius.sh first. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Xenium loader output (process_ltx_xenium_nebius.sh publishes here). +raw_dir='/scratch/task_ist_preprocessing/raw' +# scRNAseq reference is the process_ltx_sc_nebius.sh output (log_cp/hvg/pca/knn). +sc_ref="/scratch/task_ist_preprocessing/raw/ltx_human_lung_sc/ltx_full_annotation_2026_04_25/dataset.h5ad" +publish_dir='/scratch/task_ist_preprocessing/datasets' + +launch_batch() { + local params_file="$1" + local label="$2" + tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/process_datasets/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file "$params_file" \ + --config src/base/labels_nebius.config \ + --labels "task_ist_preprocessing,process_datasets,$label" +} + +cat > /tmp/params_ltx.yaml << HERE +param_list: + + - id: "ltx_human_lung_xenium_combined/B_AK-16-12103" + input_sp: "$raw_dir/ltx_human_lung_xenium/B_AK-16-12103/dataset.zarr" + input_sc: "$sc_ref" + dataset_id: "ltx_human_lung_xenium_combined/B_AK-16-12103" + dataset_name: "Human lung combined LTX Xenium B_AK-16-12103 + LTX scRNAseq" + dataset_url: "" + dataset_reference: "" + dataset_summary: "LTX Xenium human lung (sample B_AK-16-12103) + LTX human lung scRNAseq reference" + dataset_description: "LTX Xenium human lung (sample B_AK-16-12103) + LTX human lung scRNAseq reference (ltx_full_annotation_2026_04_25)" + dataset_organism: "homo_sapiens" + + - id: "ltx_human_lung_xenium_combined/L_AK14_14254" + input_sp: "$raw_dir/ltx_human_lung_xenium/L_AK14_14254/dataset.zarr" + input_sc: "$sc_ref" + dataset_id: "ltx_human_lung_xenium_combined/L_AK14_14254" + dataset_name: "Human lung combined LTX Xenium L_AK14_14254 + LTX scRNAseq" + dataset_url: "" + dataset_reference: "" + dataset_summary: "LTX Xenium human lung (sample L_AK14_14254) + LTX human lung scRNAseq reference" + dataset_description: "LTX Xenium human lung (sample L_AK14_14254) + LTX human lung scRNAseq reference (ltx_full_annotation_2026_04_25)" + dataset_organism: "homo_sapiens" + +output_sc: "\$id/output_sc.h5ad" +output_sp: "\$id/output_sp.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +launch_batch /tmp/params_ltx.yaml "ltx" diff --git a/scripts/create_resources/combine/process_datasets_mpii_nebius.sh b/scripts/create_resources/combine/process_datasets_mpii_nebius.sh new file mode 100644 index 000000000..4b714df68 --- /dev/null +++ b/scripts/create_resources/combine/process_datasets_mpii_nebius.sh @@ -0,0 +1,61 @@ +#!/bin/bash + +# Process ONLY the MPII custom human-lung Xenium dataset (combine step). +# Reads the spatial input (process_tenx_xenium loader output) from the local +# /scratch raw folder produced by process_mpii_xenium_nebius.sh, and combines it +# with the MPII human lung scRNAseq reference. +# +# The scRNAseq reference must already be standardized (scripts/create_resources/ +# sc/standardize_mpii_human_lung_sc.py) AND processed (log_cp -> hvg -> pca -> +# knn, i.e. it must carry a 'normalized' layer) before it is used here. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Xenium loader output (process_mpii_xenium_nebius.sh publishes here). +raw_dir='/scratch/task_ist_preprocessing/raw' +# scRNAseq reference is not a loader output; it lives in the S3 datasets bucket. +# Update this to wherever the standardized+processed reference was uploaded. +sc_ref="/scratch/task_ist_preprocessing/raw/mpii_human_lung_sc/current_annotation/dataset.h5ad" +publish_dir='/scratch/task_ist_preprocessing/datasets' + +launch_batch() { + local params_file="$1" + local label="$2" + tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/process_datasets/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file "$params_file" \ + --config src/base/labels_nebius.config \ + --labels "task_ist_preprocessing,process_datasets,$label" +} + +cat > /tmp/params_mpii.yaml << HERE +param_list: + + - id: "mpii_human_lung_xenium_combined/978_reg1" + input_sp: "$raw_dir/mpii_human_lung_xenium/978_reg1/dataset.zarr" + input_sc: "$sc_ref" + dataset_id: "mpii_human_lung_xenium_combined/978_reg1" + dataset_name: "Human lung combined MPII Xenium 978 reg1 + MPII scRNAseq" + dataset_url: "" + dataset_reference: "" + dataset_summary: "MPII Xenium human lung (sample 978, region 1) + MPII human lung scRNAseq reference" + dataset_description: "MPII Xenium human lung (sample 978, region 1) + MPII human lung scRNAseq reference" + dataset_organism: "homo_sapiens" + +output_sc: "\$id/output_sc.h5ad" +output_sp: "\$id/output_sp.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +launch_batch /tmp/params_mpii.yaml "mpii" diff --git a/scripts/create_resources/combine/process_datasets_nebius.sh b/scripts/create_resources/combine/process_datasets_nebius.sh new file mode 100644 index 000000000..bb634a464 --- /dev/null +++ b/scripts/create_resources/combine/process_datasets_nebius.sh @@ -0,0 +1,389 @@ +#!/bin/bash + +# TODO: The param_list metadata was mostly infered with chatGPT from the create resources scripts. +# Double check if everything's correct. + + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +input_dir="s3://openproblems-data/resources/datasets" +#publish_dir="s3://openproblems-data/resources/task_ist_preprocessing/datasets" +publish_dir='/scratch/task_ist_preprocessing/datasets' + +launch_batch() { + local params_file="$1" + local label="$2" + tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/process_datasets/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file "$params_file" \ + --config src/base/labels_nebius.config \ + --labels "task_ist_preprocessing,process_datasets,$label" +} + +# ── Batch 1: 10x Xenium ────────────────────────────────────────────────────── + +cat > /tmp/params_xenium.yaml << HERE +param_list: + + - id: "2023_10x_mouse_brain_xenium_combined/rep1" + input_sp: "$input_dir/10x_xenium/2023_10x_mouse_brain_xenium/rep1/dataset.zarr" + input_sc: "$input_dir/allen_brain_cell_atlas/2023_yao_mouse_brain_scrnaseq_10xv2/dataset.h5ad" + dataset_id: "2023_10x_mouse_brain_xenium_combined/rep1" + dataset_name: "Mouse brain combined 2023 10x Xenium rep1 2023 Yao scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/fresh-frozen-mouse-brain-replicates-1-standard" + dataset_reference: "https://www.10xgenomics.com/datasets/fresh-frozen-mouse-brain-replicates-1-standard" + dataset_summary: "Xenium V1 Fresh Frozen Mouse Brain rep1 + ABCA Mouse Brain scRNAseq" + dataset_description: "Xenium V1 Fresh Frozen Mouse Brain rep1 + ABCA Mouse Brain scRNAseq" + dataset_organism: "mus_musculus" + + - id: "2023_10x_mouse_brain_xenium_combined/rep2" + input_sp: "$input_dir/10x_xenium/2023_10x_mouse_brain_xenium/rep2/dataset.zarr" + input_sc: "$input_dir/allen_brain_cell_atlas/2023_yao_mouse_brain_scrnaseq_10xv2/dataset.h5ad" + dataset_id: "2023_10x_mouse_brain_xenium_combined/rep2" + dataset_name: "Mouse brain combined 2023 10x Xenium rep2 2023 Yao scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/fresh-frozen-mouse-brain-replicates-1-standard" + dataset_reference: "https://www.10xgenomics.com/datasets/fresh-frozen-mouse-brain-replicates-1-standard" + dataset_summary: "Xenium V1 Fresh Frozen Mouse Brain rep2 + ABCA Mouse Brain scRNAseq" + dataset_description: "Xenium V1 Fresh Frozen Mouse Brain rep2 + ABCA Mouse Brain scRNAseq" + dataset_organism: "mus_musculus" + + - id: "2023_10x_mouse_brain_xenium_combined/rep3" + input_sp: "$input_dir/10x_xenium/2023_10x_mouse_brain_xenium/rep3/dataset.zarr" + input_sc: "$input_dir/allen_brain_cell_atlas/2023_yao_mouse_brain_scrnaseq_10xv2/dataset.h5ad" + dataset_id: "2023_10x_mouse_brain_xenium_combined/rep3" + dataset_name: "Mouse brain combined 2023 10x Xenium rep3 2023 Yao scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/fresh-frozen-mouse-brain-replicates-1-standard" + dataset_reference: "https://www.10xgenomics.com/datasets/fresh-frozen-mouse-brain-replicates-1-standard" + dataset_summary: "Xenium V1 Fresh Frozen Mouse Brain rep3 + ABCA Mouse Brain scRNAseq" + dataset_description: "Xenium V1 Fresh Frozen Mouse Brain rep3 + ABCA Mouse Brain scRNAseq" + dataset_organism: "mus_musculus" + + - id: "2023_10x_human_lung_xenium_combined" + input_sp: "$input_dir/10x_xenium/2023_10x_human_lung_xenium/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2020Travaglini_human_lung_sc/dataset.h5ad" + dataset_id: "2023_10x_human_lung_xenium_combined" + dataset_name: "Human lung combined 2023 10x Xenium 2020 Travaglini scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/xenium-human-lung-preview-data-1-standard" + dataset_reference: "https://doi.org/10.1038/s41586-020-2922-4" + dataset_summary: "Xenium Preview Human Non diseased Lung FFPE + 2020 Travaglini scRNAseq" + dataset_description: "Xenium Preview Human Non diseased Lung FFPE + 2020 Travaglini scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2023_10x_human_lung_cancer_xenium_combined" + input_sp: "$input_dir/10x_xenium/2023_10x_human_lung_cancer_xenium/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "2023_10x_human_lung_cancer_xenium_combined" + dataset_name: "Human lung cancer combined 2023 10x Xenium 2024 Zuani scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/xenium-human-lung-preview-data-1-standard" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Xenium Preview Human Lung Cancer FFPE + 2024 Zuani scRNAseq" + dataset_description: "Xenium Preview Human Lung Cancer FFPE + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2024_10x_human_skin_xenium_combined" + input_sp: "$input_dir/10x_xenium/2024_10x_human_skin_xenium/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2024Ganier_human_skin_sc/dataset.h5ad" + dataset_id: "2024_10x_human_skin_xenium_combined" + dataset_name: "Human skin combined 2024 10x Xenium 2024 Ganier scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/human-skin-data-xenium-human-multi-tissue-and-cancer-panel-1-standard" + dataset_reference: "https://doi.org/10.1073/pnas.2313326120" + dataset_summary: "Xenium V1 hSkin nondiseased FFPE + 2024 Ganier scRNAseq" + dataset_description: "Xenium V1 hSkin nondiseased FFPE + 2024 Ganier scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2024_10x_human_liver_xenium_combined" + input_sp: "$input_dir/10x_xenium/2024_10x_human_liver_xenium/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2022Andrews_human_liver_sc/dataset.h5ad" + dataset_id: "2024_10x_human_liver_xenium_combined" + dataset_name: "Human liver combined 2024 10x Xenium 2022 Andrews scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/human-liver-data-xenium-human-multi-tissue-and-cancer-panel-1-standard" + dataset_reference: "https://doi.org/10.1002/hep4.1854" + dataset_summary: "Xenium V1 hLiver FFPE + 2022 Andrews scRNAseq" + dataset_description: "Xenium V1 hLiver FFPE + 2022 Andrews scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2024_10x_human_liver_cancer_xenium_combined" + input_sp: "$input_dir/10x_xenium/2024_10x_human_liver_cancer_xenium/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2022Lu_human_liver_cancer_sc/dataset.h5ad" + dataset_id: "2024_10x_human_liver_cancer_xenium_combined" + dataset_name: "Human liver cancer combined 2024 10x Xenium 2022 Lu scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/human-liver-data-xenium-human-multi-tissue-and-cancer-panel-1-standard" + dataset_reference: "https://doi.org/10.1038/s41467-022-32283-3" + dataset_summary: "Xenium V1 hLiver cancer FFPE + 2022 Lu scRNAseq" + dataset_description: "Xenium V1 hLiver cancer FFPE + 2022 Lu scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2023_10x_human_colon_cancer_xenium_combined" + input_sp: "$input_dir/10x_xenium/2023_10x_human_colon_cancer_xenium/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2020Lee_human_colon_cancer_sc/dataset.h5ad" + dataset_id: "2023_10x_human_colon_cancer_xenium_combined" + dataset_name: "Human colon cancer combined 2023 10x Xenium 2020 Lee scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/human-colon-preview-data-xenium-human-colon-gene-expression-panel-1-standard" + dataset_reference: "https://doi.org/10.1038/s41588-020-0636-z" + dataset_summary: "Xenium V1 hColon Cancer FFPE + 2020 Lee scRNAseq" + dataset_description: "Xenium V1 hColon Cancer FFPE + 2020 Lee scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2023_10x_human_breast_cancer_xenium_combined" + input_sp: "$input_dir/10x_xenium/2023_10x_human_breast_cancer_xenium/dataset.zarr" + input_sc: "$input_dir/wu_human_breast_cancer_sc/2021Wu_human_breast_cancer_sc/dataset.h5ad" + dataset_id: "2023_10x_human_breast_cancer_xenium_combined" + dataset_name: "Human breast cancer combined 2023 10x Xenium 2021 Wu scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/xenium-ffpe-human-breast-with-custom-add-on-panel-1-standard" + dataset_reference: "https://doi.org/10.1038/s41588-021-00911-1" + dataset_summary: "Xenium V1 FFPE Human Breast IDC + 2021 Wu scRNAseq" + dataset_description: "Xenium V1 FFPE Human Breast IDC + 2021 Wu scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2026_10x_human_breast_cancer_atera_combined" + input_sp: "$input_dir/10x_atera/2026_10x_human_breast_cancer_atera/dataset.zarr" + input_sc: "$input_dir/wu_human_breast_cancer_sc/2021Wu_human_breast_cancer_sc/dataset.h5ad" + dataset_id: "2026_10x_human_breast_cancer_atera_combined" + dataset_name: "Human breast cancer combined 2026 10x Atera WTA 2021 Wu scRNAseq" + dataset_url: "https://www.10xgenomics.com/datasets/atera-wta-ffpe-human-breast-cancer" + dataset_reference: "https://doi.org/10.1038/s41588-021-00911-1" + dataset_summary: "Atera WTA FFPE Human Breast Cancer (DCIS Grade 3) + 2021 Wu scRNAseq" + dataset_description: "Atera WTA FFPE Human Breast Cancer (DCIS Grade 3) + 2021 Wu scRNAseq" + dataset_organism: "homo_sapiens" + +output_sc: "\$id/output_sc.h5ad" +output_sp: "\$id/output_sp.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +launch_batch /tmp/params_xenium.yaml "xenium" + +# ── Batch 2: Vizgen MERSCOPE ────────────────────────────────────────────────── + +cat > /tmp/params_vizgen.yaml << HERE +param_list: + + - id: "2022_vizgen_human_breast_cancer_merfish_combined/rep1" + input_sp: "$input_dir/vizgen_merscope/2022_vizgen_human_breast_cancer_merfish/rep1/dataset.zarr" + input_sc: "$input_dir/wu_human_breast_cancer_sc/2021Wu_human_breast_cancer_sc/dataset.h5ad" + dataset_id: "2022_vizgen_human_breast_cancer_merfish_combined/rep1" + dataset_name: "Human breast cancer combined 2022 Vizgen MERFISH rep1 2021 Wu scRNAseq" + dataset_url: "https://info.vizgen.com/ffpe-showcase" + dataset_reference: "https://doi.org/10.1038/s41588-021-00911-1" + dataset_summary: "Vizgen Human Breast Cancer MERFISH Patient1 + 2021 Wu scRNAseq" + dataset_description: "Vizgen Human Breast Cancer MERFISH Patient1 + 2021 Wu scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2022_vizgen_human_liver_cancer_merfish_combined/rep1" + input_sp: "$input_dir/vizgen_merscope/2022_vizgen_human_liver_cancer_merfish/rep1/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2022Lu_human_liver_cancer_sc/dataset.h5ad" + dataset_id: "2022_vizgen_human_liver_cancer_merfish_combined/rep1" + dataset_name: "Human liver cancer combined 2022 Vizgen MERFISH rep1 2022 Lu scRNAseq" + dataset_url: "https://info.vizgen.com/ffpe-showcase" + dataset_reference: "https://doi.org/10.1038/s41467-022-32283-3" + dataset_summary: "Vizgen Human Liver Cancer MERFISH Patient1 + 2022 Lu scRNAseq" + dataset_description: "Vizgen Human Liver Cancer MERFISH Patient1 + 2022 Lu scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2022_vizgen_human_liver_cancer_merfish_combined/rep2" + input_sp: "$input_dir/vizgen_merscope/2022_vizgen_human_liver_cancer_merfish/rep2/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2022Lu_human_liver_cancer_sc/dataset.h5ad" + dataset_id: "2022_vizgen_human_liver_cancer_merfish_combined/rep2" + dataset_name: "Human liver cancer combined 2022 Vizgen MERFISH rep2 2022 Lu scRNAseq" + dataset_url: "https://info.vizgen.com/ffpe-showcase" + dataset_reference: "https://doi.org/10.1038/s41467-022-32283-3" + dataset_summary: "Vizgen Human Liver Cancer MERFISH Patient2 + 2022 Lu scRNAseq" + dataset_description: "Vizgen Human Liver Cancer MERFISH Patient2 + 2022 Lu scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2022_vizgen_human_lung_cancer_merfish_combined/rep1" + input_sp: "$input_dir/vizgen_merscope/2022_vizgen_human_lung_cancer_merfish/rep1/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "2022_vizgen_human_lung_cancer_merfish_combined/rep1" + dataset_name: "Human lung cancer combined 2022 Vizgen MERFISH rep1 2024 Zuani scRNAseq" + dataset_url: "https://info.vizgen.com/ffpe-showcase" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Vizgen Human Lung Cancer MERFISH Patient1 + 2024 Zuani scRNAseq" + dataset_description: "Vizgen Human Lung Cancer MERFISH Patient1 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2022_vizgen_human_lung_cancer_merfish_combined/rep2" + input_sp: "$input_dir/vizgen_merscope/2022_vizgen_human_lung_cancer_merfish/rep2/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "2022_vizgen_human_lung_cancer_merfish_combined/rep2" + dataset_name: "Human lung cancer combined 2022 Vizgen MERFISH rep2 2024 Zuani scRNAseq" + dataset_url: "https://info.vizgen.com/ffpe-showcase" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Vizgen Human Lung Cancer MERFISH Patient2 + 2024 Zuani scRNAseq" + dataset_description: "Vizgen Human Lung Cancer MERFISH Patient2 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2022_vizgen_human_colon_cancer_merfish_combined/rep1" + input_sp: "$input_dir/vizgen_merscope/2022_vizgen_human_colon_cancer_merfish/rep1/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2020Lee_human_colon_cancer_sc/dataset.h5ad" + dataset_id: "2022_vizgen_human_colon_cancer_merfish_combined/rep1" + dataset_name: "Human colon cancer combined 2022 Vizgen MERFISH rep1 2020 Lee scRNAseq" + dataset_url: "https://info.vizgen.com/ffpe-showcase" + dataset_reference: "https://doi.org/10.1038/s41588-020-0636-z" + dataset_summary: "2022 Vizgen Human Colon Cancer MERFISH Patient1 + 2020 Lee scRNAseq" + dataset_description: "2022 Vizgen Human Colon Cancer MERFISH Patient1 + 2020 Lee scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "2022_vizgen_human_colon_cancer_merfish_combined/rep2" + input_sp: "$input_dir/vizgen_merscope/2022_vizgen_human_colon_cancer_merfish/rep2/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2020Lee_human_colon_cancer_sc/dataset.h5ad" + dataset_id: "2022_vizgen_human_colon_cancer_merfish_combined/rep2" + dataset_name: "Human colon cancer combined 2022 Vizgen MERFISH rep2 2020 Lee scRNAseq" + dataset_url: "https://info.vizgen.com/ffpe-showcase" + dataset_reference: "https://doi.org/10.1038/s41588-020-0636-z" + dataset_summary: "2022 Vizgen Human Colon Cancer MERFISH Patient2 + 2020 Lee scRNAseq" + dataset_description: "2022 Vizgen Human Colon Cancer MERFISH Patient2 + 2020 Lee scRNAseq" + dataset_organism: "homo_sapiens" + +output_sc: "\$id/output_sc.h5ad" +output_sp: "\$id/output_sp.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +launch_batch /tmp/params_vizgen.yaml "vizgen" + +# ── Batch 3: Bruker CosMx ───────────────────────────────────────────────────── + +cat > /tmp/params_bruker.yaml << HERE +param_list: + + - id: "bruker_mouse_brain_cosmx_combined/rep1" + input_sp: "$input_dir/bruker_cosmx/bruker_mouse_brain_cosmx/rep1/dataset.zarr" + input_sc: "$input_dir/allen_brain_cell_atlas/2023_yao_mouse_brain_scrnaseq_10xv2/dataset.h5ad" + dataset_id: "bruker_mouse_brain_cosmx_combined/rep1" + dataset_name: "Mouse brain combined Bruker CosMx rep1 2023 Yao scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/cosmx-smi-mouse-brain-ffpe-dataset/" + dataset_reference: "10.1038/s41586-023-06812-z" + dataset_summary: "Bruker CosMx Mouse Brain + ABCA Mouse Brain scRNAseq" + dataset_description: "Bruker CosMx Mouse Brain + ABCA Mouse Brain scRNAseq" + dataset_organism: "mus_musculus" + + - id: "bruker_human_liver_cosmx_combined" + input_sp: "$input_dir/bruker_cosmx/bruker_human_liver_cosmx/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2022Andrews_human_liver_sc/dataset.h5ad" + dataset_id: "bruker_human_liver_cosmx_combined" + dataset_name: "Human liver combined Bruker CosMx 2022 Andrews scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/human-liver-rna-ffpe-dataset/" + dataset_reference: "https://doi.org/10.1002/hep4.1854" + dataset_summary: "Bruker CosMx Human Liver + 2022 Andrews scRNAseq" + dataset_description: "Bruker CosMx Human Liver + 2022 Andrews scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "bruker_human_liver_cancer_cosmx_combined" + input_sp: "$input_dir/bruker_cosmx/bruker_human_liver_cancer_cosmx/dataset.zarr" + input_sc: "$input_dir/scrnaseq_for_ist/2022Lu_human_liver_cancer_sc/dataset.h5ad" + dataset_id: "bruker_human_liver_cancer_cosmx_combined" + dataset_name: "Human liver cancer combined Bruker CosMx 2022 Lu scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/human-liver-rna-ffpe-dataset/" + dataset_reference: "https://doi.org/10.1038/s41467-022-32283-3" + dataset_summary: "Bruker CosMx Human Liver Cancer + 2022 Lu scRNAseq" + dataset_description: "Bruker CosMx Human Liver Cancer + 2022 Lu scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "bruker_human_lung_cancer_cosmx_combined/lung5_rep1" + input_sp: "$input_dir/bruker_cosmx/bruker_human_lung_cancer_cosmx/lung5_rep1/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "bruker_human_lung_cancer_cosmx_combined/lung5_rep1" + dataset_name: "Human lung cancer combined Bruker CosMx Lung5 rep1 2024 Zuani scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung5 Rep1 + 2024 Zuani scRNAseq" + dataset_description: "Bruker CosMx Human Lung Cancer Lung5 Rep1 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "bruker_human_lung_cancer_cosmx_combined/lung5_rep2" + input_sp: "$input_dir/bruker_cosmx/bruker_human_lung_cancer_cosmx/lung5_rep2/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "bruker_human_lung_cancer_cosmx_combined/lung5_rep2" + dataset_name: "Human lung cancer combined Bruker CosMx Lung5 rep2 2024 Zuani scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung5 Rep2 + 2024 Zuani scRNAseq" + dataset_description: "Bruker CosMx Human Lung Cancer Lung5 Rep2 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "bruker_human_lung_cancer_cosmx_combined/lung5_rep3" + input_sp: "$input_dir/bruker_cosmx/bruker_human_lung_cancer_cosmx/lung5_rep3/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "bruker_human_lung_cancer_cosmx_combined/lung5_rep3" + dataset_name: "Human lung cancer combined Bruker CosMx Lung5 rep3 2024 Zuani scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung5 Rep3 + 2024 Zuani scRNAseq" + dataset_description: "Bruker CosMx Human Lung Cancer Lung5 Rep3 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "bruker_human_lung_cancer_cosmx_combined/lung6" + input_sp: "$input_dir/bruker_cosmx/bruker_human_lung_cancer_cosmx/lung6/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "bruker_human_lung_cancer_cosmx_combined/lung6" + dataset_name: "Human lung cancer combined Bruker CosMx Lung6 2024 Zuani scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung6 + 2024 Zuani scRNAseq" + dataset_description: "Bruker CosMx Human Lung Cancer Lung6 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "bruker_human_lung_cancer_cosmx_combined/lung9_rep1" + input_sp: "$input_dir/bruker_cosmx/bruker_human_lung_cancer_cosmx/lung9_rep1/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "bruker_human_lung_cancer_cosmx_combined/lung9_rep1" + dataset_name: "Human lung cancer combined Bruker CosMx Lung9 rep1 2024 Zuani scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung9 Rep1 + 2024 Zuani scRNAseq" + dataset_description: "Bruker CosMx Human Lung Cancer Lung9 Rep1 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "bruker_human_lung_cancer_cosmx_combined/lung9_rep2" + input_sp: "$input_dir/bruker_cosmx/bruker_human_lung_cancer_cosmx/lung9_rep2/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "bruker_human_lung_cancer_cosmx_combined/lung9_rep2" + dataset_name: "Human lung cancer combined Bruker CosMx Lung9 rep2 2024 Zuani scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung9 Rep2 + 2024 Zuani scRNAseq" + dataset_description: "Bruker CosMx Human Lung Cancer Lung9 Rep2 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "bruker_human_lung_cancer_cosmx_combined/lung12" + input_sp: "$input_dir/bruker_cosmx/bruker_human_lung_cancer_cosmx/lung12/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "bruker_human_lung_cancer_cosmx_combined/lung12" + dataset_name: "Human lung cancer combined Bruker CosMx Lung12 2024 Zuani scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung12 + 2024 Zuani scRNAseq" + dataset_description: "Bruker CosMx Human Lung Cancer Lung12 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + + - id: "bruker_human_lung_cancer_cosmx_combined/lung13" + input_sp: "$input_dir/bruker_cosmx/bruker_human_lung_cancer_cosmx/lung13/dataset.zarr" + input_sc: "$input_dir/zuani_human_nsclc_sc/2024Zuani_human_nsclc_sc/dataset.h5ad" + dataset_id: "bruker_human_lung_cancer_cosmx_combined/lung13" + dataset_name: "Human lung cancer combined Bruker CosMx Lung13 2024 Zuani scRNAseq" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_reference: "https://www.ebi.ac.uk/biostudies/arrayexpress/studies/E-MTAB-13526" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung13 + 2024 Zuani scRNAseq" + dataset_description: "Bruker CosMx Human Lung Cancer Lung13 + 2024 Zuani scRNAseq" + dataset_organism: "homo_sapiens" + +output_sc: "\$id/output_sc.h5ad" +output_sp: "\$id/output_sp.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +launch_batch /tmp/params_bruker.yaml "bruker" diff --git a/scripts/create_resources/sc/process_allen_brain_cell_atlas_brain_nebius.sh b/scripts/create_resources/sc/process_allen_brain_cell_atlas_brain_nebius.sh new file mode 100644 index 000000000..bb1093bdb --- /dev/null +++ b/scripts/create_resources/sc/process_allen_brain_cell_atlas_brain_nebius.sh @@ -0,0 +1,59 @@ +#!/bin/bash + +# Nebius version of process_allen_brain_cell_atlas_brain.sh: process the ABCA (2023 Yao) +# mouse-brain scRNAseq reference on Nebius and publish to the scratch raw/ folder (like the +# other *_nebius.sh scripts), so the scratch-reading combine scripts (e.g. +# process_datasets_bruker_nebius.sh) find it at +# /scratch/task_ist_preprocessing/raw/allen_brain_cell_atlas/2023_yao_mouse_brain_scrnaseq_10xv2/dataset.h5ad +# The S3 sibling (process_allen_brain_cell_atlas_brain.sh) publishes to resources/datasets. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +publish_dir="/scratch/task_ist_preprocessing/raw" + +cat > /tmp/params_allen_brain_sc.yaml << HERE +param_list: + + - id: allen_brain_cell_atlas/2023_yao_mouse_brain_scrnaseq_10xv2 + regions: + - CTXsp + - HPF + - HY + - Isocortex + - MB + - OLF + - TH + dataset_name: ABCA Mouse Brain scRNAseq + dataset_url: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE246717 + dataset_reference: 10.1038/s41586-023-06812-z + dataset_summary: A high-resolution scRNAseq atlas of cell types in the whole mouse brain + dataset_description: See dataset_reference for more information. Note that we only took the 10xv2 data from the dataset. + dataset_organism: mus_musculus + +sample_n_obs: 500000 +sample_obs_weight: subclass +sample_transform: log +sample_seed: 42 +keep_files: false # disk isn't large enough + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_allen_brain_cell_atlas/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_allen_brain_sc.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,allen_brain_cell_atlas diff --git a/scripts/create_resources/sc/process_ganier_human_skin_sc_nebius.sh b/scripts/create_resources/sc/process_ganier_human_skin_sc_nebius.sh new file mode 100644 index 000000000..d1e3dd084 --- /dev/null +++ b/scripts/create_resources/sc/process_ganier_human_skin_sc_nebius.sh @@ -0,0 +1,46 @@ +#!/bin/bash + +# Process the Ganier human-skin scRNAseq reference on Nebius. +# +# Unlike the custom ltx/mpii SC scripts (which standardize a raw counts file +# locally and push it through the generic process_scrnaseq workflow), Ganier has +# a self-contained dedicated workflow: it downloads via its loader and runs +# hvg -> pca -> knn -> extract_uns_metadata, emitting a full file_common_scrnaseq +# dataset (with a 'normalized' layer, HVG, PCA and kNN) directly. +# +# The dataset source and all metadata defaults live in the workflow config +# (src/datasets/workflows/process_ganier_human_skin_sc/config.vsh.yaml), so only +# the id needs to be specified here. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +publish_dir="/scratch/task_ist_preprocessing/raw" + +cat > /tmp/params_ganier_sc.yaml << HERE +param_list: + + - id: scrnaseq_for_ist/2024Ganier_human_skin_sc + +keep_files: false + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_ganier_human_skin_sc/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_ganier_sc.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,ganier_human_skin_sc diff --git a/scripts/create_resources/sc/process_kuppe_kidney_sc_nebius.sh b/scripts/create_resources/sc/process_kuppe_kidney_sc_nebius.sh new file mode 100644 index 000000000..2be154794 --- /dev/null +++ b/scripts/create_resources/sc/process_kuppe_kidney_sc_nebius.sh @@ -0,0 +1,48 @@ +#!/bin/bash + +# Process the Kuppe kidney snRNA-seq references (cancer + healthy) through the +# generic SC workflow (log_cp -> hvg -> pca -> knn -> extract_uns_metadata), +# turning the standardized raw counts files into file_common_scrnaseq datasets +# (with a 'normalized' layer, HVG, PCA and kNN) usable as input_sc for the +# corresponding process_datasets_*_nebius.sh combine step. +# +# Prerequisite: run scripts/create_resources/sc/standardize_kuppe_kidney_sc.py +# locally on each Flex-snRNA/{Cancer,Healthy}/annotated.h5ad and upload its output +# to the S3 paths referenced below (see that script's docstring). + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# standardized (raw, counts-only) SC references produced by standardize_kuppe_kidney_sc.py +raw_dir="s3://openproblems-data/resources/raw_data/txSim_custom/kuppe_kidney" +publish_dir="/scratch/task_ist_preprocessing/raw" + +cat > /tmp/params_kuppe_kidney_sc.yaml << HERE +param_list: + + - id: "kuppe_kidney_cancer_sc/annotated" + input: "$raw_dir/cancer/dataset.h5ad" + + - id: "kuppe_kidney_healthy_sc/annotated" + input: "$raw_dir/healthy/dataset.h5ad" + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_scrnaseq/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_kuppe_kidney_sc.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,kuppe_kidney_sc diff --git a/scripts/create_resources/sc/process_ltx_sc_nebius.sh b/scripts/create_resources/sc/process_ltx_sc_nebius.sh new file mode 100644 index 000000000..1dc3042d7 --- /dev/null +++ b/scripts/create_resources/sc/process_ltx_sc_nebius.sh @@ -0,0 +1,44 @@ +#!/bin/bash + +# Process the LTX human-lung scRNAseq reference through the generic SC workflow +# (log_cp -> hvg -> pca -> knn -> extract_uns_metadata), turning the standardized +# raw counts file into a file_common_scrnaseq dataset (with a 'normalized' layer, +# HVG, PCA and kNN) usable as input_sc for process_datasets_ltx_nebius.sh. +# +# Prerequisite: run scripts/create_resources/sc/standardize_ltx_sc.py locally and +# upload its output to the S3 path referenced by `input` below (the same path the +# standardize script's docstring uploads to). + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# standardized (raw, counts-only) SC reference produced by standardize_ltx_sc.py +input_h5ad="s3://openproblems-data/resources/raw_data/txSim_custom/ltx/dataset.h5ad" +publish_dir="/scratch/task_ist_preprocessing/raw" + +cat > /tmp/params_ltx_sc.yaml << HERE +param_list: + + - id: "ltx_human_lung_sc/ltx_full_annotation_2026_04_25" + input: "$input_h5ad" + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_scrnaseq/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_ltx_sc.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,ltx_sc diff --git a/scripts/create_resources/sc/process_mpii_human_lung_sc_nebius.sh b/scripts/create_resources/sc/process_mpii_human_lung_sc_nebius.sh new file mode 100644 index 000000000..4c5915660 --- /dev/null +++ b/scripts/create_resources/sc/process_mpii_human_lung_sc_nebius.sh @@ -0,0 +1,43 @@ +#!/bin/bash + +# Process the MPII human-lung scRNAseq reference through the generic SC workflow +# (log_cp -> hvg -> pca -> knn -> extract_uns_metadata), turning the standardized +# raw counts file into a file_common_scrnaseq dataset (with a 'normalized' layer, +# HVG, PCA and kNN) usable as input_sc for process_datasets_mpii_nebius.sh. +# +# Prerequisite: run scripts/create_resources/sc/standardize_mpii_human_lung_sc.py +# locally and upload its output to the S3 path referenced by `input` below. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# standardized (raw, counts-only) SC reference produced by standardize_mpii_human_lung_sc.py +input_h5ad="s3://openproblems-data/resources/raw_data/txSim_custom/MPII/dataset.h5ad" +publish_dir="/scratch/task_ist_preprocessing/raw" + +cat > /tmp/params_mpii_sc.yaml << HERE +param_list: + + - id: "mpii_human_lung_sc/current_annotation" + input: "$input_h5ad" + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_scrnaseq/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_mpii_sc.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,mpii_sc diff --git a/scripts/create_resources/sc/standardize_kuppe_kidney_sc.py b/scripts/create_resources/sc/standardize_kuppe_kidney_sc.py new file mode 100644 index 000000000..c5600fda9 --- /dev/null +++ b/scripts/create_resources/sc/standardize_kuppe_kidney_sc.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python +"""Standardize a Kuppe kidney Flex-snRNA ``annotated.h5ad`` into the OpenProblems +SC *loader-output* format (raw, counts-only). + +The Kuppe kidney MERFISH release ships two snRNA-seq references under +``Flex-snRNA/{Cancer,Healthy}/annotated.h5ad``. They use DIFFERENT cell-type +vocabularies and gene sets, so this script standardizes ONE file per run — point +it at each in turn to get two independent references (as chosen for this dataset). + +Input (per file): ``annotated.h5ad`` with + - raw integer counts in ``layers['counts']`` (``X`` / ``layers['log1p_norm']`` + are already normalized — ignored here; the ``normalized`` layer is + recomputed downstream), + - gene SYMBOLS in ``var.index`` and ENSEMBL ids in ``var['gene_ids']``, + - the curated cell-type label in ``obs['annotation']`` (Cancer: 13 classes + incl. ``Tumor`` and some empty-string entries; Healthy: 19 kidney classes). + ``celltype_hint`` / ``predictions_unconstrained`` are automated ontology + hints, NOT the curated label — deliberately not used. +Output: a clean ``dataset.h5ad`` mirroring what the SC loaders emit (a ``counts`` + layer, ``cell_type`` in ``.obs``, ``feature_name`` (symbols) in ``.var`` and + ``dataset_*`` metadata in ``.uns``). + +Deliberately NOT computed here (added downstream by process_scrnaseq: +log_cp -> hvg -> pca -> knn): the ``normalized`` layer, HVG flags, PCA, kNN. + +Memory: the source files also carry ``X``, a ``log1p_norm`` layer and large +``obsp``/``uns`` (neighbour graph, leiden markers). We therefore read ONLY +``layers['counts']`` + ``obs`` + ``var`` via ``anndata.io.read_elem``, so peak +memory ~= the counts matrix. + +Run locally with an env that has anndata>=0.11 + scipy, e.g. (both conditions):: + + PY=/opt/miniconda3/envs/spatialdata/bin/python + BASE="/Volumes/SeagateHHD/Kuppe_kidney_merfish/240824_download/Flex-snRNA" + + $PY scripts/create_resources/sc/standardize_kuppe_kidney_sc.py \ + --condition cancer \ + --input "$BASE/Cancer/annotated.h5ad" \ + --output ~/Downloads/kuppe_kidney_cancer_sc/dataset.h5ad + + $PY scripts/create_resources/sc/standardize_kuppe_kidney_sc.py \ + --condition healthy \ + --input "$BASE/Healthy/annotated.h5ad" \ + --output ~/Downloads/kuppe_kidney_healthy_sc/dataset.h5ad + +Then upload (profile 'op'), matching the paths in process_kuppe_kidney_sc_nebius.sh:: + + aws s3 cp --profile op ~/Downloads/kuppe_kidney_cancer_sc/dataset.h5ad \ + s3://openproblems-data/resources/raw_data/txSim_custom/kuppe_kidney/cancer/dataset.h5ad + aws s3 cp --profile op ~/Downloads/kuppe_kidney_healthy_sc/dataset.h5ad \ + s3://openproblems-data/resources/raw_data/txSim_custom/kuppe_kidney/healthy/dataset.h5ad +""" + +import argparse +import gc +from pathlib import Path + +import anndata as ad +import h5py +import numpy as np +import pandas as pd +from scipy.sparse import csr_matrix + +try: + from anndata.io import read_elem +except ImportError: # older anndata + from anndata.experimental import read_elem + +# ── Per-condition metadata presets ────────────────────────────────────────── +# One reference per condition (chosen because the two files use disjoint cell-type +# vocabularies). Every field is overridable on the CLI. +PRESETS = { + "cancer": dict( + dataset_id="kuppe_kidney_cancer_sc", + dataset_name="Kuppe kidney cancer snRNA-seq reference (Flex)", + dataset_summary=( + "Kuppe lab kidney cancer single-nucleus RNA-seq reference (10x Flex), " + "used as the annotation reference for the Kuppe kidney MERFISH dataset." + ), + dataset_description=( + "Single-nucleus RNA-seq reference of kidney cancer tissue (Kuppe lab, 10x " + "Flex), curated cell-type label in obs['annotation'] (incl. a Tumor class). " + "Used to annotate the Kuppe kidney MERFISH spatial data." + ), + disease="kidney cancer", + ), + "healthy": dict( + dataset_id="kuppe_kidney_healthy_sc", + dataset_name="Kuppe kidney healthy snRNA-seq reference (Flex)", + dataset_summary=( + "Kuppe lab healthy kidney single-nucleus RNA-seq reference (10x Flex), " + "used as the annotation reference for the Kuppe kidney MERFISH dataset." + ), + dataset_description=( + "Single-nucleus RNA-seq reference of healthy kidney tissue (Kuppe lab, 10x " + "Flex), curated kidney cell-type label in obs['annotation']. Used to annotate " + "the Kuppe kidney MERFISH spatial data." + ), + disease="normal", + ), +} + +DATASET_ORGANISM = "homo_sapiens" + +# placeholder/missing cell-type strings to drop (the similarity metric forbids +# NaN/"None" in cell_type; the Cancer file has empty-string annotations) +_MISSING = {"nan", "none", "na", "", "unknown"} + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--input", required=True, help="Path to the Kuppe kidney annotated.h5ad") + ap.add_argument("--output", required=True, help="Path to write the standardized dataset.h5ad") + ap.add_argument( + "--condition", + choices=sorted(PRESETS), + help="Fills dataset metadata + disease from a preset (cancer/healthy). Optional if all --dataset-* are given.", + ) + ap.add_argument("--celltype-col", default="annotation", help="obs column holding the curated cell type (default: annotation)") + ap.add_argument("--counts-layer", default="counts", help="layer holding raw integer counts (default: counts)") + # metadata overrides (default to the preset when --condition is given) + ap.add_argument("--dataset-id") + ap.add_argument("--dataset-name") + ap.add_argument("--dataset-summary") + ap.add_argument("--dataset-description") + ap.add_argument("--dataset-url", default="") + ap.add_argument("--dataset-reference", default="") + ap.add_argument("--disease") + ap.add_argument("--organism", default=DATASET_ORGANISM) + args = ap.parse_args() + + preset = PRESETS.get(args.condition, {}) + + # CLI override wins, else preset value. + dataset_id = args.dataset_id or preset.get("dataset_id") + dataset_name = args.dataset_name or preset.get("dataset_name") + dataset_summary = args.dataset_summary or preset.get("dataset_summary") + dataset_description = args.dataset_description or preset.get("dataset_description") + disease = args.disease or preset.get("disease") + if not (dataset_id and dataset_name and dataset_summary and dataset_description): + raise SystemExit( + "Missing dataset metadata: pass --condition {cancer,healthy} or all of " + "--dataset-id/--dataset-name/--dataset-summary/--dataset-description." + ) + + inp = Path(args.input).expanduser() + print(f"Reading (selectively) {inp} ...", flush=True) + with h5py.File(inp, "r") as f: + layers = f.get("layers", {}) + if args.counts_layer not in layers: + raise SystemExit(f"layers['{args.counts_layer}'] not found in {inp} (have: {list(layers.keys())})") + counts = read_elem(f["layers"][args.counts_layer]) + obs = read_elem(f["obs"]) + var = read_elem(f["var"]) + print(f" loaded counts {counts.shape} + obs{obs.shape} + var{var.shape}", flush=True) + + # ── counts: enforce raw, non-negative, integer, csr float32 ────────────── + counts = counts.tocsr() if hasattr(counts, "tocsr") else csr_matrix(counts) + counts = counts.astype("float32") + if counts.min() < 0: + raise SystemExit(f"layers['{args.counts_layer}'] contains negative values — not raw counts") + frac_int = float(np.mean(counts.data == np.round(counts.data))) if counts.nnz else 1.0 + print(f" counts: nnz={counts.nnz:,} min={counts.min()} max={counts.max()} integer-fraction={frac_int:.3f}", flush=True) + # Downstream count-based methods (RCTD, SPLIT) call spacexr::check_counts, which + # hard-rejects non-integer references. The Kuppe X/log1p_norm layers ARE normalized; + # only layers['counts'] is raw — guard against pointing at the wrong layer. + if frac_int < 0.999: + raise SystemExit( + f"layers['{args.counts_layer}'] is not raw integer counts " + f"(integer-fraction={frac_int:.3f}, max={counts.max():.4g}) — looks normalized. " + f"Point --counts-layer at the raw count matrix." + ) + + # ── var: feature_name (symbols, = index) + feature_id (ENSEMBL) ────────── + var_index = var.index.astype(str) + new_var = pd.DataFrame(index=var_index) + new_var["feature_name"] = var_index.values + new_var["gene_symbol"] = var_index.values + new_var["feature_id"] = ( + var["gene_ids"].astype(str).values if "gene_ids" in var.columns else var_index.values + ) + + # ── obs: cell_type (from the curated annotation) + disease ─────────────── + if args.celltype_col not in obs.columns: + raise SystemExit(f"obs['{args.celltype_col}'] not found — cannot set cell_type (have: {list(obs.columns)[:30]})") + new_obs = pd.DataFrame(index=obs.index.astype(str)) + new_obs["cell_type"] = obs[args.celltype_col].astype(str).values + if disease: + new_obs["disease"] = disease + + del obs, var + gc.collect() + + # ── assemble clean AnnData (X and layers['counts'] share the matrix) ───── + adata = ad.AnnData(X=counts, obs=new_obs, var=new_var) + adata.layers["counts"] = counts + + # similarity metric forbids NaN / "None" / empty in cell_type — drop those cells + bad = adata.obs["cell_type"].str.strip().str.lower().isin(_MISSING) | adata.obs["cell_type"].isna() + if bool(bad.any()): + print(f" dropping {int(bad.sum())} cells with missing/placeholder cell_type", flush=True) + adata = adata[~bad.values].copy() + + # cell_type must be categorical (similarity reads .dtype.categories) + adata.obs["cell_type"] = pd.Categorical(adata.obs["cell_type"].astype(str)) + if "disease" in adata.obs.columns: + adata.obs["disease"] = pd.Categorical(adata.obs["disease"].astype(str)) + + # ── dataset metadata ───────────────────────────────────────────────────── + adata.uns["dataset_id"] = dataset_id + adata.uns["dataset_name"] = dataset_name + adata.uns["dataset_url"] = args.dataset_url + adata.uns["dataset_reference"] = args.dataset_reference + adata.uns["dataset_summary"] = dataset_summary + adata.uns["dataset_description"] = dataset_description + adata.uns["dataset_organism"] = args.organism + + out = Path(args.output).expanduser() + out.parent.mkdir(parents=True, exist_ok=True) + print( + f"Writing {out}\n" + f" {adata.n_obs:,} cells x {adata.n_vars:,} genes | layers={list(adata.layers)} | " + f"cell_type classes={adata.obs['cell_type'].cat.categories.size} | dataset_id={dataset_id}", + flush=True, + ) + adata.write_h5ad(out, compression="gzip") + print("Done.", flush=True) + + +if __name__ == "__main__": + main() diff --git a/scripts/create_resources/sc/standardize_ltx_sc.py b/scripts/create_resources/sc/standardize_ltx_sc.py new file mode 100644 index 000000000..514ff9e9c --- /dev/null +++ b/scripts/create_resources/sc/standardize_ltx_sc.py @@ -0,0 +1,174 @@ +#!/usr/bin/env python +"""Standardize the custom LTX human-lung scRNAseq annotation into the +OpenProblems SC *loader-output* format (raw, counts-only). + +Input : ltx_full_annotation_2026_04_25.h5ad (363,099 cells x 18,117 genes; + gene SYMBOLS in var index, ENSEMBL in var['gene_ids']; raw counts in + layers['counts']; 25-class annotation in obs['celltype_final']). +Output: a clean dataset.h5ad mirroring what the SC loaders emit (a `counts` + layer, `cell_type` in .obs, `feature_name` (symbols) in .var, and + `dataset_*` metadata in .uns). + +Deliberately NOT computed here (added downstream by process_scrnaseq: +log_cp -> hvg -> pca -> knn): the `normalized` layer, HVG flags, PCA, kNN. + +Memory: the source file also carries X, a normcounts layer and a large obsp +(neighbour graph). We therefore read ONLY layers['counts'] + obs + var via +anndata.io.read_elem, so peak memory ~= the counts matrix, not the full 20 GB. + +Run locally with an env that has anndata>=0.11 + scipy, e.g.:: + + /opt/miniconda3/envs/spatialdata/bin/python \ + scripts/create_resources/sc/standardize_ltx_sc.py \ + --input ~/projects/tests_fot_txsim/ltx_full_annotation_2026_04_25.h5ad \ + --output ~/projects/tests_fot_txsim/ltx_human_lung_sc/dataset.h5ad + +Then upload (profile 'op'):: + + aws s3 cp --profile op \ + s3://openproblems-data/resources/raw_data/txSim_custom/ltx/ltx_human_lung_sc_raw.h5ad +""" + +import argparse +import gc +from pathlib import Path + +import anndata as ad +import h5py +import numpy as np +import pandas as pd +from scipy.sparse import csr_matrix + +try: + from anndata.io import read_elem +except ImportError: # older anndata + from anndata.experimental import read_elem + +# ── Dataset metadata ──────────────────────────────────────────────────────── +DATASET_ID = "ltx_human_lung_sc" +DATASET_NAME = "LTX human lung scRNAseq reference (full annotation 2026-04-25)" +DATASET_URL = "" # custom / unpublished LTX reference +DATASET_REFERENCE = "" +DATASET_SUMMARY = ( + "Custom LTX human lung scRNAseq reference with a 25-class cell-type " + "annotation (celltype_final), used as the annotation reference for the " + "LTX Xenium human lung samples." +) +DATASET_DESCRIPTION = ( + "Single-cell RNA-seq reference of human lung (LTX cohort), used as the " + "annotation reference for the LTX Xenium dataset. 363,099 cells x 18,117 " + "genes; primary label in celltype_final (25 classes)." +) +DATASET_ORGANISM = "homo_sapiens" + +# primary label (per user): cell_type <- celltype_final. The coarser LTX columns +# ('lineage', 'celltype coarse', 'celltype refined') could optionally be mapped +# to cell_type_level2..4 (all optional in file_common_scrnaseq); left out here to +# avoid guessing granularity — add if the benchmark needs a hierarchy. +CELLTYPE_COL = "celltype_final" +COUNTS_LAYER = "counts" + +# optional metadata carried over from .obs if present (standardized -> source) +EXTRA_OBS = { + "donor_id": "sampleID", + "sample": "Sample Pseudonym", + "sample_type": "sample type", + "batch": "batch", + "lineage": "lineage", +} + +_MISSING = {"nan", "none", "na", "", "unknown"} + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument("--input", required=True, help="Path to the raw LTX annotation h5ad") + ap.add_argument("--output", required=True, help="Path to write the standardized dataset.h5ad") + args = ap.parse_args() + + inp = Path(args.input).expanduser() + print(f"Reading (selectively) {inp} ...", flush=True) + with h5py.File(inp, "r") as f: + if COUNTS_LAYER not in f.get("layers", {}): + raise SystemExit(f"layers['{COUNTS_LAYER}'] not found in {inp}") + counts = read_elem(f["layers"][COUNTS_LAYER]) + obs = read_elem(f["obs"]) + var = read_elem(f["var"]) + print(f" loaded counts {counts.shape} + obs{obs.shape} + var{var.shape}", flush=True) + + # ── counts: enforce raw, non-negative, csr float32 ─────────────────────── + counts = counts.tocsr() if hasattr(counts, "tocsr") else csr_matrix(counts) + counts = counts.astype("float32") + if counts.min() < 0: + raise SystemExit(f"layers['{COUNTS_LAYER}'] contains negative values — not raw counts") + frac_int = float(np.mean(counts.data == np.round(counts.data))) if counts.nnz else 1.0 + print(f" counts: nnz={counts.nnz:,} min={counts.min()} max={counts.max()} integer-fraction={frac_int:.3f}", flush=True) + # Enforce raw integer counts. Downstream count-based methods (RCTD, SPLIT) call + # spacexr::check_counts, which hard-rejects non-integer references. A processed + # atlas export can carry log-normalized values in layers['counts'] (and in .raw.X) + # while retaining only per-cell totals in obs — do NOT let that pass silently as a + # "reference". If this fires, point COUNTS_LAYER at the true raw count matrix. + if frac_int < 0.999: + raise SystemExit( + f"layers['{COUNTS_LAYER}'] is not raw integer counts " + f"(integer-fraction={frac_int:.3f}, max={counts.max():.4g}) — looks normalized/" + f"log-transformed. RCTD/SPLIT require integer counts; standardize needs the raw " + f"count matrix, not this layer." + ) + + # ── var: feature_name (symbols, = index) + feature_id (ENSEMBL) ────────── + var_index = var.index.astype(str) + new_var = pd.DataFrame(index=var_index) + new_var["feature_name"] = var_index.values + new_var["gene_symbol"] = var_index.values + new_var["feature_id"] = ( + var["gene_ids"].astype(str).values if "gene_ids" in var.columns else var_index.values + ) + + # ── obs: cell_type (from celltype_final) + carried metadata ────────────── + if CELLTYPE_COL not in obs.columns: + raise SystemExit(f"obs['{CELLTYPE_COL}'] not found — cannot set cell_type") + new_obs = pd.DataFrame(index=obs.index.astype(str)) + new_obs["cell_type"] = obs[CELLTYPE_COL].astype(str).values + for new, src in EXTRA_OBS.items(): + if src in obs.columns: + new_obs[new] = obs[src].astype(str).values + + del obs, var + gc.collect() + + # ── assemble clean AnnData (X and layers['counts'] share the matrix) ───── + adata = ad.AnnData(X=counts, obs=new_obs, var=new_var) + adata.layers["counts"] = counts + + # similarity metric forbids NaN / "None" in cell_type — drop those cells + bad = adata.obs["cell_type"].str.strip().str.lower().isin(_MISSING) | adata.obs["cell_type"].isna() + if bool(bad.any()): + print(f" dropping {int(bad.sum())} cells with missing/placeholder cell_type", flush=True) + adata = adata[~bad.values].copy() + + adata.obs["cell_type"] = pd.Categorical(adata.obs["cell_type"].astype(str)) + + # ── dataset metadata ───────────────────────────────────────────────────── + adata.uns["dataset_id"] = DATASET_ID + adata.uns["dataset_name"] = DATASET_NAME + adata.uns["dataset_url"] = DATASET_URL + adata.uns["dataset_reference"] = DATASET_REFERENCE + adata.uns["dataset_summary"] = DATASET_SUMMARY + adata.uns["dataset_description"] = DATASET_DESCRIPTION + adata.uns["dataset_organism"] = DATASET_ORGANISM + + out = Path(args.output).expanduser() + out.parent.mkdir(parents=True, exist_ok=True) + print( + f"Writing {out}\n" + f" {adata.n_obs:,} cells x {adata.n_vars:,} genes | layers={list(adata.layers)} | " + f"cell_type classes={adata.obs['cell_type'].cat.categories.size}", + flush=True, + ) + adata.write_h5ad(out, compression="gzip") + print("Done.", flush=True) + + +if __name__ == "__main__": + main() diff --git a/scripts/create_resources/sc/standardize_mpii_human_lung_sc.py b/scripts/create_resources/sc/standardize_mpii_human_lung_sc.py new file mode 100644 index 000000000..3d433f224 --- /dev/null +++ b/scripts/create_resources/sc/standardize_mpii_human_lung_sc.py @@ -0,0 +1,214 @@ +#!/usr/bin/env python +"""Standardize the MPII custom human-lung scRNAseq annotation into the +OpenProblems SC *loader-output* format. + +Input : the pre-annotated ``current_annotation.h5ad`` (101,141 cells x 20,497 + genes, ENSEMBL var index, HLCA-style cell-type hierarchy). +Output: a clean ``dataset.h5ad`` mirroring what the SC loaders emit + (e.g. ``wu_human_breast_cancer_sc``): a ``counts`` layer, a ``cell_type`` + hierarchy in ``.obs``, ``feature_name`` (gene symbols) in ``.var`` and + ``dataset_*`` metadata in ``.uns``. + +Deliberately NOT computed here (added by the downstream SC process step +loader -> log_cp -> hvg -> pca -> knn): ``normalized`` layer, HVG flags, PCA, +kNN graph. This file is the equivalent of the loader's ``output_raw``. + +Cell-type mapping (per user choice): + cell_type <- celltype (finest, 67 classes) + cell_type_level2 <- celltype_l2 (21 classes) + cell_type_level3 <- celltype_l3 (41 classes) + cell_type_level4 <- celltype_l4 (67 classes) + cell_type_level1 <- celltype_l1 (4 classes, kept for reference) + +Run locally with an env that has anndata + scipy, e.g.:: + + /opt/miniconda3/envs/spatialdata/bin/python \ + scripts/create_resources/sc/standardize_mpii_human_lung_sc.py \ + --input ~/Downloads/current_annotation.h5ad \ + --output ~/Downloads/mpii_human_lung_sc/dataset.h5ad + +Then upload, e.g.:: + + aws s3 cp ~/Downloads/mpii_human_lung_sc/dataset.h5ad \ + s3://openproblems-data/resources/datasets/mpii_human_lung_sc/current_annotation/dataset.h5ad +""" + +import argparse +import gc +from pathlib import Path + +import anndata as ad +import numpy as np +import pandas as pd +from scipy.sparse import csr_matrix + +# ── Dataset metadata ──────────────────────────────────────────────────────── +DATASET_ID = "mpii_human_lung_sc" +DATASET_NAME = "MPII human lung scRNAseq reference (current annotation)" +DATASET_URL = "" # custom / unpublished MPII reference +DATASET_REFERENCE = "" +DATASET_SUMMARY = ( + "Custom MPII human lung scRNAseq reference with a fine-grained " + "(67 cell types) HLCA-style annotation hierarchy." +) +DATASET_DESCRIPTION = ( + "Custom single-cell RNA-seq reference of healthy human lung generated at MPII, " + "used as the annotation reference for the MPII Xenium human lung dataset. " + "101,141 cells x 20,497 genes with a 4-level cell-type hierarchy (celltype_l1..l4)." +) +DATASET_ORGANISM = "homo_sapiens" + +# standardized cell-type column <- source column in current_annotation.h5ad +CELLTYPE_MAP = { + "cell_type": "celltype", # primary label (finest, == celltype_l4) — user choice + "cell_type_level1": "celltype_l1", + "cell_type_level2": "celltype_l2", + "cell_type_level3": "celltype_l3", + "cell_type_level4": "celltype_l4", +} + +# extra metadata carried over from .obs if present (standardized -> source) +EXTRA_OBS = { + "donor_id": "mpii_subject_id", + "sample": "sample", + "assay": "assay", + "sex": "sex", + "disease": "disease", + "tissue": "tissue", + "organism": "organism", +} + +_MISSING = {"nan", "None", "NA", "NaN", "", "unknown", "Unknown"} + + +def _symbols_from_var(var: pd.DataFrame, index: pd.Index) -> np.ndarray: + """Gene symbols for .var: prefer gene_names/gene_name, fall back to ENSEMBL id.""" + symbols = None + for col in ("gene_names", "gene_name"): + if col in var.columns: + symbols = pd.Series(var[col].astype(str).values, index=range(len(var))) + break + ensembl = ( + var["gene_ids"].astype(str).values + if "gene_ids" in var.columns + else index.astype(str).values + ) + if symbols is None: + symbols = pd.Series(index.astype(str).values, index=range(len(var))) + # never leave a null/placeholder feature_name — fall back to the ENSEMBL id + symbols = symbols.replace({m: np.nan for m in {"nan", "None", "NA", "NaN", ""}}) + symbols = symbols.fillna(pd.Series(ensembl, index=range(len(var)))) + return symbols.values, ensembl + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument( + "--input", + default=str(Path.home() / "Downloads" / "current_annotation.h5ad"), + help="Path to current_annotation.h5ad", + ) + ap.add_argument( + "--output", + default=str(Path.home() / "Downloads" / "mpii_human_lung_sc" / "dataset.h5ad"), + help="Path to write the standardized dataset.h5ad", + ) + ap.add_argument( + "--counts-from", + choices=["X", "raw"], + default="X", + help="Which matrix holds the raw counts (both are integer in this file; X is aligned to .var/.obs).", + ) + args = ap.parse_args() + + print(f"Reading {args.input} ...", flush=True) + adata = ad.read_h5ad(args.input) + print(f" loaded: {adata.n_obs} cells x {adata.n_vars} genes", flush=True) + + # ── counts ────────────────────────────────────────────────────────────── + if args.counts_from == "raw": + if adata.raw is None: + raise SystemExit("--counts-from raw requested but .raw is absent") + counts = adata.raw.X + var_src = adata.raw.var + else: + counts = adata.X + var_src = adata.var + counts = counts.tocsr() if hasattr(counts, "tocsr") else csr_matrix(counts) + counts = counts.astype("float32") + if counts.min() < 0: + raise SystemExit("counts contain negative values — X is likely not raw counts; try --counts-from raw") + + # ── var: feature_name (symbols) + feature_id (ENSEMBL) ──────────────────── + symbols, ensembl = _symbols_from_var(var_src, adata.var_names) + var = pd.DataFrame(index=adata.var_names.astype(str)) + var["feature_name"] = symbols + var["gene_symbol"] = symbols + var["feature_id"] = ensembl + + # ── obs: cell_type hierarchy + carried-over metadata ───────────────────── + obs = pd.DataFrame(index=adata.obs_names.astype(str)) + for new, src in CELLTYPE_MAP.items(): + if src in adata.obs.columns: + obs[new] = adata.obs[src].astype(str).values + else: + print(f" WARNING: source obs column '{src}' not found; skipping '{new}'") + for new, src in EXTRA_OBS.items(): + if src in adata.obs.columns and new not in obs.columns: + obs[new] = adata.obs[src].astype(str).values + + if "cell_type" not in obs.columns: + raise SystemExit("primary source column 'celltype' not found — cannot set cell_type") + + # free the large original (incl. .raw, obsm, obsp) before building the output + del adata + gc.collect() + + # ── assemble clean AnnData ─────────────────────────────────────────────── + new = ad.AnnData(X=counts.copy(), obs=obs, var=var) + new.layers["counts"] = counts + + # similarity metric forbids NaN / "None" in cell_type — drop those cells + bad = new.obs["cell_type"].str.strip().str.lower().isin({m.lower() for m in _MISSING}) + bad = bad | new.obs["cell_type"].isna() + if bool(bad.any()): + print(f" dropping {int(bad.sum())} cells with missing/placeholder cell_type", flush=True) + new = new[~bad.values].copy() + + # cell-type columns must be categorical (similarity reads .dtype.categories) + for col in [c for c in new.obs.columns if c.startswith("cell_type")]: + new.obs[col] = pd.Categorical(new.obs[col].astype(str)) + + # ── dataset metadata ───────────────────────────────────────────────────── + new.uns["dataset_id"] = DATASET_ID + new.uns["dataset_name"] = DATASET_NAME + new.uns["dataset_url"] = DATASET_URL + new.uns["dataset_reference"] = DATASET_REFERENCE + new.uns["dataset_summary"] = DATASET_SUMMARY + new.uns["dataset_description"] = DATASET_DESCRIPTION + new.uns["dataset_organism"] = DATASET_ORGANISM + + out = Path(args.output).expanduser() + out.parent.mkdir(parents=True, exist_ok=True) + print( + f"Writing {out}\n" + f" {new.n_obs} cells x {new.n_vars} genes | layers={list(new.layers)} | " + f"cell_type classes={new.obs['cell_type'].cat.categories.size}", + flush=True, + ) + new.write_h5ad(out, compression="gzip") + print("Done.", flush=True) + print( + "\nNext steps (this file is the RAW, counts-only reference):\n" + f" 1. Upload: aws s3 cp {out} " + "s3://openproblems-data/resources/raw_data/txSim_custom/MPII/mpii_human_lung_sc_raw.h5ad\n" + " 2. Process: run scripts/create_resources/sc/process_mpii_human_lung_sc_nebius.sh\n" + " (generic datasets/workflows/process_scrnaseq: log_cp -> hvg -> pca -> knn).\n" + " It adds the 'normalized' layer, HVG, PCA and kNN and publishes the final\n" + f" input_sc to s3://openproblems-data/resources/datasets/{DATASET_ID}/current_annotation/dataset.h5ad\n" + " 3. Combine: run scripts/create_resources/combine/process_datasets_mpii_nebius.sh" + ) + + +if __name__ == "__main__": + main() diff --git a/scripts/create_resources/spatial/process_bruker_cosmx_nsclc_nebius.sh b/scripts/create_resources/spatial/process_bruker_cosmx_nsclc_nebius.sh new file mode 100644 index 000000000..10943f92d --- /dev/null +++ b/scripts/create_resources/spatial/process_bruker_cosmx_nsclc_nebius.sh @@ -0,0 +1,116 @@ +#!/bin/bash + +# Get the root of the repository +REPO_ROOT=$(git rev-parse --show-toplevel) + +# Ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# store the loader output locally, mirroring the process_datasets layout ($id/) +# under the sibling raw/ folder (same as the other *_nebius.sh spatial scripts) +publish_dir="/scratch/task_ist_preprocessing/raw" + +# Raw NSCLC archives mirrored to S3 (see scripts/create_resources/spatial/mirror_bruker_to_s3.sh). +# Each sample ships two tar.gz archives: the flat files + cell labels, and the raw morphology images. +raw_dir="s3://openproblems-data/resources/raw_data/bruker_cosmx" + +cat > /tmp/params.yaml << HERE +param_list: + + - id: "bruker_cosmx/bruker_human_lung_cancer_cosmx/lung5_rep1" + input_raw: "$raw_dir/Lung5_Rep1+SMI+Flat+data.tar.gz" + input_morphology: "$raw_dir/Lung5_Rep1+RawMorphologyImages.tar.gz" + dataset_name: "Bruker CosMx Human Lung Cancer Lung5 Rep1" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung5 Rep1 dataset on FFPE." + dataset_description: "Bruker CosMx Human Lung Cancer Lung5 Rep1 dataset on FFPE. Adenocarcinoma, G1, T2aN2M0, IIIA, 75% tumour content." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "bruker_cosmx/bruker_human_lung_cancer_cosmx/lung5_rep2" + input_raw: "$raw_dir/Lung5_Rep2+SMI+Flat+data.tar.gz" + input_morphology: "$raw_dir/Lung5_Rep2+RawMorphologyImages.tar.gz" + dataset_name: "Bruker CosMx Human Lung Cancer Lung5 Rep2" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung5 Rep2 dataset on FFPE." + dataset_description: "Bruker CosMx Human Lung Cancer Lung5 Rep2 dataset on FFPE. Adenocarcinoma, G1, T2aN2M0, IIIA, 75% tumour content." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "bruker_cosmx/bruker_human_lung_cancer_cosmx/lung5_rep3" + input_raw: "$raw_dir/Lung5_Rep3+SMI+Flat+data.tar.gz" + input_morphology: "$raw_dir/Lung5_Rep3+RawMorphologyImages.tar.gz" + dataset_name: "Bruker CosMx Human Lung Cancer Lung5 Rep3" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung5 Rep3 dataset on FFPE." + dataset_description: "Bruker CosMx Human Lung Cancer Lung5 Rep3 dataset on FFPE. Adenocarcinoma, G1, T2aN2M0, IIIA, 75% tumour content." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "bruker_cosmx/bruker_human_lung_cancer_cosmx/lung6" + input_raw: "$raw_dir/Lung6+SMI+Flat+data.tar.gz" + input_morphology: "$raw_dir/Lung6+RawMorphologyImages.tar.gz" + dataset_name: "Bruker CosMx Human Lung Cancer Lung6" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung6 dataset on FFPE." + dataset_description: "Bruker CosMx Human Lung Cancer Lung6 dataset on FFPE. Squamous cell carcinoma, G2, T2bN2M0, IIIA, 90% tumour content." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "bruker_cosmx/bruker_human_lung_cancer_cosmx/lung9_rep1" + input_raw: "$raw_dir/Lung9_Rep1+SMI+Flat+data.tar.gz" + input_morphology: "$raw_dir/Lung9_Rep1+RawMorphologyImages.tar.gz" + dataset_name: "Bruker CosMx Human Lung Cancer Lung9 Rep1" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung9 Rep1 dataset on FFPE." + dataset_description: "Bruker CosMx Human Lung Cancer Lung9 Rep1 dataset on FFPE. Adenocarcinoma, G3, T3N1M0, IIIA, 65% tumour content." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "bruker_cosmx/bruker_human_lung_cancer_cosmx/lung9_rep2" + input_raw: "$raw_dir/Lung9_Rep2+SMI+Flat+data.tar.gz" + input_morphology: "$raw_dir/Lung9_Rep2+RawMorphologyImages.tar.gz" + dataset_name: "Bruker CosMx Human Lung Cancer Lung9 Rep2" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung9 Rep2 dataset on FFPE." + dataset_description: "Bruker CosMx Human Lung Cancer Lung9 Rep2 dataset on FFPE. Adenocarcinoma, G3, T3N1M0, IIIA, 65% tumour content." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "bruker_cosmx/bruker_human_lung_cancer_cosmx/lung12" + input_raw: "$raw_dir/Lung12+SMI+Flat+data.tar.gz" + input_morphology: "$raw_dir/Lung12+RawMorphologyImages.tar.gz" + dataset_name: "Bruker CosMx Human Lung Cancer Lung12" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung12 dataset on FFPE." + dataset_description: "Bruker CosMx Human Lung Cancer Lung12 dataset on FFPE. Adenocarcinoma, G3, T4N0M0, IIIA, 85% tumour content." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "bruker_cosmx/bruker_human_lung_cancer_cosmx/lung13" + input_raw: "$raw_dir/Lung13+SMI+Flat+data.tar.gz" + input_morphology: "$raw_dir/Lung13+RawMorphologyImages.tar.gz" + dataset_name: "Bruker CosMx Human Lung Cancer Lung13" + dataset_url: "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" + dataset_summary: "Bruker CosMx Human Lung Cancer Lung13 dataset on FFPE." + dataset_description: "Bruker CosMx Human Lung Cancer Lung13 dataset on FFPE. Adenocarcinoma, G1, T3N0M0, IIB, 55% tumour content." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + +output_dataset: "\$id/dataset.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_bruker_cosmx_nsclc/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,bruker_cosmx_nsclc diff --git a/scripts/create_resources/spatial/process_kuppe_kidney_merfish_nebius.sh b/scripts/create_resources/spatial/process_kuppe_kidney_merfish_nebius.sh new file mode 100644 index 000000000..3a1862d4c --- /dev/null +++ b/scripts/create_resources/spatial/process_kuppe_kidney_merfish_nebius.sh @@ -0,0 +1,72 @@ +#!/bin/bash + +# Convert the Kuppe kidney MERFISH (Vizgen MERSCOPE) samples into standardized SpatialData +# dataset.zarr via the generic process_vizgen_merscope workflow, publishing the loader output +# to the local /scratch raw folder (like the other *_nebius.sh spatial scripts). +# +# Inputs are the 2D (z3-only) region_0 copies mirrored to S3 by +# scripts/create_resources/spatial/upload_kuppe_kidney_merfish_2d.sh. The vizgen_merscope loader +# reads only z-plane 3 (spatialdata_io.merscope(z_layers=3)), so the z3-only copy is complete for +# this pipeline. Each region_0 already ships a Vizgen-native cell_boundaries.parquet +# (GeoParquet, ZIndex 0..6) which spatialdata_io reads directly (ZIndex==0 slice). +# +# NOTE: both samples have >65535 cell boundaries (cancer ~303k, healthy ~399k at ZIndex 0), so +# this exercises the loader's chunked-rasterization path — make sure the launch uses a +# vizgen_merscope image built AFTER that fix reached build/main (--revision build/main --pull-latest). +# +# The downstream combine step must read its spatial inputs from the same publish_dir +# (set input_dir="/scratch/task_ist_preprocessing/raw" there), pairing each spatial sample with the +# matching Kuppe SC reference (kuppe_kidney_{cancer,healthy}_sc, see process_kuppe_kidney_sc_nebius.sh). + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# 2D region_0 copies produced by upload_kuppe_kidney_merfish_2d.sh +raw_dir="s3://openproblems-data/resources/raw_data/txSim_custom/kuppe_kidney/merfish" +# store the loader output locally, mirroring the process_datasets layout ($id/) +publish_dir="/scratch/task_ist_preprocessing/raw" + +cat > /tmp/params_kuppe_kidney_merfish.yaml << HERE +param_list: + + - id: "vizgen_merscope/kuppe_kidney_cancer_merfish/rep1" + dataset_id: "vizgen_merscope/kuppe_kidney_cancer_merfish/rep1" + input: "$raw_dir/cancer/region_0" + dataset_name: "Kuppe Kidney Cancer MERFISH" + dataset_url: "" + dataset_reference: "" + dataset_summary: "Kuppe lab Clear Cell Kidney Carcinoma MERFISH (Vizgen MERSCOPE, 500-gene Pan-Cancer panel)." + dataset_description: "Clear Cell Kidney Carcinoma FFPE sample profiled with MERFISH on the Vizgen MERSCOPE using the 500-gene Pan-Cancer panel (Kuppe lab). Cells segmented with Cellpose Cyto2 on the cell-boundary stain. Paired with the kuppe_kidney_cancer_sc snRNA-seq reference. Note: the Pan-Cancer panel does not separate NKT vs T cells or pericytes vs vSMC." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "vizgen_merscope/kuppe_kidney_healthy_merfish/rep1" + dataset_id: "vizgen_merscope/kuppe_kidney_healthy_merfish/rep1" + input: "$raw_dir/healthy/region_0" + dataset_name: "Kuppe Kidney Healthy MERFISH" + dataset_url: "" + dataset_reference: "" + dataset_summary: "Kuppe lab healthy kidney MERFISH (Vizgen MERSCOPE, 140-gene Spapros panel)." + dataset_description: "Healthy kidney FFPE sample profiled with MERFISH on the Vizgen MERSCOPE using a 140-gene panel designed with Spapros from Kuppe et al 2021 snRNA-seq data. Cells segmented with Cellpose Cyto2 on the cell-boundary stain. Paired with the kuppe_kidney_healthy_sc snRNA-seq reference." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + +output_dataset: "\$id/dataset.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_vizgen_merscope/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_kuppe_kidney_merfish.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,kuppe_kidney_merfish diff --git a/scripts/create_resources/spatial/process_ltx_xenium_nebius.sh b/scripts/create_resources/spatial/process_ltx_xenium_nebius.sh new file mode 100644 index 000000000..b6d61da96 --- /dev/null +++ b/scripts/create_resources/spatial/process_ltx_xenium_nebius.sh @@ -0,0 +1,69 @@ +#!/bin/bash + +# Process the custom LTX human-lung Xenium dataset (spatial side) — TWO samples. +# Reuses the generic 10x Xenium loader (process_tenx_xenium workflow) to convert +# each raw Xenium OUTPUT DIRECTORY (stored unzipped in the S3 raw_data bucket) +# into a standardized SpatialData dataset.zarr, published to the local /scratch +# raw folder (mirroring the layout process_datasets_*.sh reads from). +# +# Unlike the MPII run (which pointed at a single .zip), the LTX samples are stored +# as unzipped Xenium output directories, so `input` is the S3 directory prefix. +# The tenx_xenium loader handles a directory input: it walks the staged tree to +# find cell_feature_matrix.h5 and reads it with spatialdata_io.xenium(). +# +# Each sample carries a multi-channel morphology_focus (ch0=DAPI); the loader keeps +# it as `morphology_mip`, which process_dataset later renames to the API-required +# `image`. Writing that multiscale image requires spatialdata>=0.8.0 in the loader +# container (older spatialdata + zarr>=3.2 fails with a rectilinear chunk-grid +# error). See the tenx_atera loader for the proven pin. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# store the loader output locally, mirroring the process_datasets layout ($id/) +# but under a sibling raw/ folder +publish_dir="/scratch/task_ist_preprocessing/raw" + +# raw Xenium output directories in the S3 raw_data bucket (unzipped) +ltx_base="s3://openproblems-data/resources/raw_data/txSim_custom/ltx" + +cat > /tmp/params_ltx_xenium.yaml << HERE +param_list: + + - id: "ltx_human_lung_xenium/B_AK-16-12103" + input: "$ltx_base/B_AK-16-12103" + dataset_name: "LTX human lung Xenium (sample B_AK-16-12103)" + dataset_url: "" + dataset_summary: "Custom LTX Xenium In Situ run of human lung tissue (sample B_AK-16-12103)." + dataset_description: "Custom Xenium In Situ human lung dataset (LTX cohort, sample B_AK-16-12103). Paired with the LTX human lung scRNAseq annotation reference (ltx_full_annotation_2026_04_25)." + dataset_organism: "homo_sapiens" + segmentation_id: [cell, nucleus] + + - id: "ltx_human_lung_xenium/L_AK14_14254" + input: "$ltx_base/L_AK14_14254" + dataset_name: "LTX human lung Xenium (sample L_AK14_14254)" + dataset_url: "" + dataset_summary: "Custom LTX Xenium In Situ run of human lung tissue (sample L_AK14_14254)." + dataset_description: "Custom Xenium In Situ human lung dataset (LTX cohort, sample L_AK14_14254). Paired with the LTX human lung scRNAseq annotation reference (ltx_full_annotation_2026_04_25)." + dataset_organism: "homo_sapiens" + segmentation_id: [cell, nucleus] + +output_dataset: "\$id/dataset.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_tenx_xenium/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_ltx_xenium.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,ltx_xenium diff --git a/scripts/create_resources/spatial/process_mpii_xenium_nebius.sh b/scripts/create_resources/spatial/process_mpii_xenium_nebius.sh new file mode 100644 index 000000000..1fcc55613 --- /dev/null +++ b/scripts/create_resources/spatial/process_mpii_xenium_nebius.sh @@ -0,0 +1,55 @@ +#!/bin/bash + +# Process ONLY the MPII custom human-lung Xenium dataset (spatial side). +# Converts the raw Xenium output zip (stored in the S3 raw_data bucket) into the +# standardized SpatialData dataset.zarr via the process_tenx_xenium workflow, +# and publishes it to the local /scratch raw folder (mirroring the layout that +# process_datasets_mpii_nebius.sh reads from). +# +# NOTE: this is a genuine Xenium run (instrument XETG00117). If the run fails +# with an irregular / rectilinear chunk-grid error while reading the loader +# output, switch the --main-script below to the tenx_atera workflow +# (target/nextflow/datasets/workflows/process_tenx_atera/main.nf), whose loader +# carries the rechunk_uniform() fix for exactly that issue. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# store the loader output locally, mirroring the process_datasets layout ($id/) +# but under a sibling raw/ folder +publish_dir="/scratch/task_ist_preprocessing/raw" + +# raw Xenium output zip in the S3 raw_data bucket +input_zip="s3://openproblems-data/resources/raw_data/txSim_custom/MPII/978_reg1_output-XETG00117__0015978__Region_1__20240718__175145.zip" + +cat > /tmp/params_mpii_xenium.yaml << HERE +param_list: + + - id: "mpii_human_lung_xenium/978_reg1" + input: "$input_zip" + dataset_name: "MPII human lung Xenium (sample 978, region 1)" + dataset_url: "" + dataset_summary: "Custom MPII Xenium In Situ run of human lung tissue (sample 978, region 1, XETG00117, 2024-07-18)." + dataset_description: "Custom Xenium In Situ human lung dataset generated at MPII (sample 978, region 1, instrument XETG00117, acquired 2024-07-18). Paired with the MPII human lung scRNAseq annotation reference." + dataset_organism: "homo_sapiens" + segmentation_id: [cell, nucleus] + +output_dataset: "\$id/dataset.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_tenx_xenium/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_mpii_xenium.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,mpii_xenium diff --git a/scripts/create_resources/spatial/process_vizgen_merscope_nebius.sh b/scripts/create_resources/spatial/process_vizgen_merscope_nebius.sh new file mode 100644 index 000000000..cde29366e --- /dev/null +++ b/scripts/create_resources/spatial/process_vizgen_merscope_nebius.sh @@ -0,0 +1,132 @@ +#!/bin/bash + +# Nebius variant of process_vizgen_merscope.sh: converts the raw Vizgen MERSCOPE FFPE +# showcase datasets into standardized SpatialData dataset.zarr via the process_vizgen_merscope +# workflow, and publishes the loader output to the local /scratch raw folder (like the other +# *_nebius.sh spatial scripts) instead of s3://openproblems-data/resources/datasets. +# +# The downstream combine step (scripts/create_resources/combine/process_datasets_vizgen_nebius.sh) +# must therefore read its spatial inputs from the same location: set its +# input_dir="/scratch/task_ist_preprocessing/raw" +# (it currently points at s3://openproblems-data/resources/datasets). +# +# NOTE 1 (fixes): the vizgen_merscope loader now carries two fixes that only take effect once +# committed and rebuilt onto build/main (this launch uses --revision build/main --pull-latest): +# - read_boundary_hdf5 no longer hardcodes zIndex_3 (falls back to any available z-plane, +# logs n_skipped) so it no longer drops/KeyErrors on cells absent at z=3; +# - the >65535-cell chunked rasterization merges correctly (was silently a no-op before). +# NOTE 2 (auth): inputs are staged from s3://openproblems-data/resources/raw_data/txSim_custom/ +# vizgen_merscope (the same bucket the Nebius env reads for every other spatial dataset), NOT +# directly from gs://vz-ffpe-showcase. The showcase bucket is Vizgen's access-controlled GCS +# bucket (an anonymous GET returns "Anonymous caller does not have storage.objects.get access"), +# and the Nebius compute env has no Google Cloud credentials, so staging gs:// paths failed as an +# anonymous caller before the loader ever ran. A z3-only (2D, lossless-for-this-pipeline) copy is +# mirrored to S3 by scripts/create_resources/spatial/upload_vizgen_merscope_2d.sh — run that once +# (from a machine authenticated to both clouds) before launching this. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# store the loader output locally, mirroring the process_datasets layout ($id/) +# under the sibling raw/ folder (same as the other *_nebius.sh spatial scripts) +publish_dir="/scratch/task_ist_preprocessing/raw" + +# raw showcase inputs, mirrored from gs://vz-ffpe-showcase to S3 (see NOTE 2 above and +# scripts/create_resources/spatial/upload_vizgen_merscope_2d.sh) +viz_base="s3://openproblems-data/resources/raw_data/txSim_custom/vizgen_merscope" + +cat > /tmp/params_vizgen_merscope.yaml << HERE +param_list: + + - id: "vizgen_merscope/2022_vizgen_human_breast_cancer_merfish/rep1" + dataset_id: "vizgen_merscope/2022_vizgen_human_breast_cancer_merfish/rep1" + input: "$viz_base/HumanBreastCancerPatient1" + dataset_name: "Vizgen Human Breast Cancer MERFISH Patient1" + dataset_url: "https://info.vizgen.com/ffpe-showcase?submissionGuid=a93dbab5-c128-4269-afe3-82ea2bf9cdaf" + dataset_summary: "Human Breast Cancer data from the MERSCOPE FFPE Human Immuno-Oncology Data Release." + dataset_description: "The MERSCOPE FFPE Human Immuno-Oncology Data Release was generated using the MERSCOPE FFPE Sample Prep Solution and the MERSCOPE Immuno-Oncology Predesigned Panel. This data release includes 16 MERFISH datasets generated by the MERSCOPE Platform from 8 different human tumor types, each measuring 500 genes representing approximately 4 billion transcripts and 9 million cells cumulatively." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "vizgen_merscope/2022_vizgen_human_liver_cancer_merfish/rep1" + dataset_id: "vizgen_merscope/2022_vizgen_human_liver_cancer_merfish/rep1" + input: "$viz_base/HumanLiverCancerPatient1" + dataset_name: "Vizgen Human Liver Cancer MERFISH Patient1" + dataset_url: "https://info.vizgen.com/ffpe-showcase?submissionGuid=a93dbab5-c128-4269-afe3-82ea2bf9cdaf" + dataset_summary: "Human Liver Cancer data from the MERSCOPE FFPE Human Immuno-Oncology Data Release." + dataset_description: "The MERSCOPE FFPE Human Immuno-Oncology Data Release was generated using the MERSCOPE FFPE Sample Prep Solution and the MERSCOPE Immuno-Oncology Predesigned Panel. This data release includes 16 MERFISH datasets generated by the MERSCOPE Platform from 8 different human tumor types, each measuring 500 genes representing approximately 4 billion transcripts and 9 million cells cumulatively." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "vizgen_merscope/2022_vizgen_human_liver_cancer_merfish/rep2" + dataset_id: "vizgen_merscope/2022_vizgen_human_liver_cancer_merfish/rep2" + input: "$viz_base/HumanLiverCancerPatient2" + dataset_name: "Vizgen Human Liver Cancer MERFISH Patient2" + dataset_url: "https://info.vizgen.com/ffpe-showcase?submissionGuid=a93dbab5-c128-4269-afe3-82ea2bf9cdaf" + dataset_summary: "Human Liver Cancer data from the MERSCOPE FFPE Human Immuno-Oncology Data Release." + dataset_description: "The MERSCOPE FFPE Human Immuno-Oncology Data Release was generated using the MERSCOPE FFPE Sample Prep Solution and the MERSCOPE Immuno-Oncology Predesigned Panel. This data release includes 16 MERFISH datasets generated by the MERSCOPE Platform from 8 different human tumor types, each measuring 500 genes representing approximately 4 billion transcripts and 9 million cells cumulatively." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "vizgen_merscope/2022_vizgen_human_lung_cancer_merfish/rep1" + dataset_id: "vizgen_merscope/2022_vizgen_human_lung_cancer_merfish/rep1" + input: "$viz_base/HumanLungCancerPatient1" + dataset_name: "Vizgen Human Lung Cancer MERFISH Patient1" + dataset_url: "https://info.vizgen.com/ffpe-showcase?submissionGuid=a93dbab5-c128-4269-afe3-82ea2bf9cdaf" + dataset_summary: "Human Lung Cancer data from the MERSCOPE FFPE Human Immuno-Oncology Data Release." + dataset_description: "The MERSCOPE FFPE Human Immuno-Oncology Data Release was generated using the MERSCOPE FFPE Sample Prep Solution and the MERSCOPE Immuno-Oncology Predesigned Panel. This data release includes 16 MERFISH datasets generated by the MERSCOPE Platform from 8 different human tumor types, each measuring 500 genes representing approximately 4 billion transcripts and 9 million cells cumulatively." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "vizgen_merscope/2022_vizgen_human_lung_cancer_merfish/rep2" + dataset_id: "vizgen_merscope/2022_vizgen_human_lung_cancer_merfish/rep2" + input: "$viz_base/HumanLungCancerPatient2" + dataset_name: "Vizgen Human Lung Cancer MERFISH Patient2" + dataset_url: "https://info.vizgen.com/ffpe-showcase?submissionGuid=a93dbab5-c128-4269-afe3-82ea2bf9cdaf" + dataset_summary: "Human Lung Cancer data from the MERSCOPE FFPE Human Immuno-Oncology Data Release." + dataset_description: "The MERSCOPE FFPE Human Immuno-Oncology Data Release was generated using the MERSCOPE FFPE Sample Prep Solution and the MERSCOPE Immuno-Oncology Predesigned Panel. This data release includes 16 MERFISH datasets generated by the MERSCOPE Platform from 8 different human tumor types, each measuring 500 genes representing approximately 4 billion transcripts and 9 million cells cumulatively." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "vizgen_merscope/2022_vizgen_human_colon_cancer_merfish/rep1" + dataset_id: "vizgen_merscope/2022_vizgen_human_colon_cancer_merfish/rep1" + input: "$viz_base/HumanColonCancerPatient1" + dataset_name: "2022 Vizgen Human Colon Cancer MERFISH Patient1" + dataset_url: "https://info.vizgen.com/ffpe-showcase" + dataset_summary: "Human Colon Cancer data from the MERSCOPE FFPE Human Immuno-Oncology Data Release." + dataset_description: "The MERSCOPE FFPE Human Immuno-Oncology Data Release was generated using the MERSCOPE FFPE Sample Prep Solution and the MERSCOPE Immuno-Oncology Predesigned Panel. It includes datasets from various human tumor types, each measuring 500 genes representing approximately 4 billion transcripts and 9 million cells cumulatively." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + + - id: "vizgen_merscope/2022_vizgen_human_colon_cancer_merfish/rep2" + dataset_id: "vizgen_merscope/2022_vizgen_human_colon_cancer_merfish/rep2" + input: "$viz_base/HumanColonCancerPatient2" + dataset_name: "2022 Vizgen Human Colon Cancer MERFISH Patient2" + dataset_url: "https://info.vizgen.com/ffpe-showcase" + dataset_summary: "Human Colon Cancer data from the MERSCOPE FFPE Human Immuno-Oncology Data Release." + dataset_description: "Same as above." + dataset_organism: "homo_sapiens" + segmentation_id: ["cell"] + +output_dataset: "\$id/dataset.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +# Additional patients (melanoma / ovarian / prostate / uterine) are available commented-out in +# the s3 sibling scripts/create_resources/spatial/process_vizgen_merscope.sh + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_vizgen_merscope/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_vizgen_merscope.yaml \ + --config src/base/labels_nebius.config \ + --labels datasets,vizgen_merscope diff --git a/scripts/create_resources/spatial/upload_kuppe_kidney_merfish_2d.sh b/scripts/create_resources/spatial/upload_kuppe_kidney_merfish_2d.sh new file mode 100644 index 000000000..24f6e9ce6 --- /dev/null +++ b/scripts/create_resources/spatial/upload_kuppe_kidney_merfish_2d.sh @@ -0,0 +1,81 @@ +#!/bin/bash + +# Upload a 2D-only copy of the Kuppe kidney MERFISH (Vizgen MERSCOPE) raw data to S3. +# +# The raw region_0 folders carry 7 focal z-planes per channel (mosaic_*_z0..z6.tif), +# ~500-820 GB per sample. The repo's vizgen_merscope loader reads ONLY z-plane 3 +# (spatialdata_io.merscope(..., z_layers=3), which globs the images dir for stains and +# opens mosaic__z3.tif). So we mirror a z3-only copy: lossless for this 2D +# pipeline, ~79 GB (cancer) + ~135 GB (healthy). +# +# Kept per sample (region_0/): mosaic_<5 stains>_z3.tif, images/manifest.json, +# images/micron_to_mosaic_pixel_transform.csv, boundary_mask_images/boundaries_z3.tif, +# detected_transcripts.csv (all molecules), cell_by_gene.csv, cell_metadata.csv, +# cell_boundaries.parquet, summary.png. +# Dropped: every mosaic_*_z{0,1,2,4,5,6}.tif and boundaries_z{0,1,2,4,5,6}.tif. +# +# Idempotent + resumable: each file is skipped if already in S3 with a matching size, +# so just re-run if the connection drops. aws s3 cp handles multipart for the big tifs. +# +# Usage: +# SRC_BASE=/Volumes/SeagateHHD/Kuppe_kidney_merfish/240824_download/merfish \ +# S3_DEST=s3://openproblems-data/resources/raw_data/txSim_custom/kuppe_kidney/merfish \ +# AWS_PROFILE=op ./upload_kuppe_kidney_merfish_2d.sh + +set -euo pipefail + +SRC_BASE="${SRC_BASE:-/Volumes/SeagateHHD/Kuppe_kidney_merfish/240824_download/merfish}" +S3_DEST="${S3_DEST:-s3://openproblems-data/resources/raw_data/txSim_custom/kuppe_kidney/merfish}" +AWS_PROFILE="${AWS_PROFILE:-op}" +SAMPLES=(${SAMPLES:-cancer healthy}) +Z="${Z:-3}" # which single z-plane to keep + +export AWS_PROFILE +command -v aws >/dev/null || { echo "ERROR: aws CLI not found" >&2; exit 1; } + +bucket="$(echo "$S3_DEST" | sed -E 's#^s3://([^/]+)/.*#\1#')" +prefix="$(echo "$S3_DEST" | sed -E 's#^s3://[^/]+/(.*)#\1#')" + +file_size() { stat -f%z "$1" 2>/dev/null || stat -c%s "$1"; } +s3_size() { aws s3api head-object --bucket "$1" --key "$2" --query 'ContentLength' --output text 2>/dev/null || true; } + +# Build the wanted-file list for a sample's region_0 (relative paths under region_0/). +wanted_rel() { + local r="$SRC_BASE/$1/region_0" + ( cd "$r" && \ + ls images/mosaic_*_z${Z}.tif \ + images/manifest.json \ + images/micron_to_mosaic_pixel_transform.csv \ + boundary_mask_images/boundaries_z${Z}.tif \ + detected_transcripts.csv cell_by_gene.csv cell_metadata.csv \ + cell_boundaries.parquet summary.png 2>/dev/null ) +} + +total_files=0; uploaded=0; skipped=0 +for sample in "${SAMPLES[@]}"; do + echo "================================================================" + echo "Sample: $sample ($SRC_BASE/$sample/region_0)" + while IFS= read -r rel; do + [ -z "$rel" ] && continue + src="$SRC_BASE/$sample/region_0/$rel" + key="$prefix/$sample/region_0/$rel" + total_files=$((total_files+1)) + local_sz="$(file_size "$src")" + remote_sz="$(s3_size "$bucket" "$key")" + if [[ -n "$remote_sz" && "$remote_sz" != "None" && "$remote_sz" == "$local_sz" ]]; then + printf ' SKIP %-55s (%s bytes, already in S3)\n' "$rel" "$local_sz" + skipped=$((skipped+1)) + continue + fi + printf ' UP %-55s (%.2f GB)\n' "$rel" "$(awk -v b="$local_sz" 'BEGIN{print b/1e9}')" + aws s3 cp "$src" "s3://$bucket/$key" --only-show-errors + uploaded=$((uploaded+1)) + done < <(wanted_rel "$sample") +done + +echo "================================================================" +echo "Done. files=$total_files uploaded=$uploaded skipped=$skipped" +echo "Loader --input paths:" +for sample in "${SAMPLES[@]}"; do + echo " $S3_DEST/$sample/region_0" +done diff --git a/scripts/create_test_resources/2021_wu_human_breast_cancer_scrnaseq.sh b/scripts/create_test_resources/2021_wu_human_breast_cancer_scrnaseq.sh new file mode 100644 index 000000000..4e24b687d --- /dev/null +++ b/scripts/create_test_resources/2021_wu_human_breast_cancer_scrnaseq.sh @@ -0,0 +1,88 @@ +#!/bin/bash + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Local (nextflow run) variant of 2021_wu_human_breast_cancer_scrnaseq_nebius.sh. +# Matching single-cell reference for the atera human breast-cancer spatial test sample. +# Publishes to the local resources_test/ instead of scratch. +# +# The process_wu_human_breast_cancer_sc workflow does NOT subsample internally, so this is +# a two-step chain (as in the nebius script): +# 1. run the workflow -> full processed SC atlas under resources_test/common +# 2. chain the openproblems `subsample` module -> ~400-cell test reference +# `nextflow run` is synchronous, so step 2 runs only after step 1 finishes (no --wait needed). +publish_dir="resources_test/common" + +full_id="2021_wu_human_breast_cancer_scrnaseq_full" +test_id="2021_wu_human_breast_cancer_scrnaseq" + +# Cap per-process CPU requests to the 8 cores available on this machine. Component +# labels otherwise request up to highcpu=30, which exceeds the machine and the +# Nextflow local executor would then fail to schedule those processes. +cat > /tmp/labels_8cpu.config << 'HERE' +process { + withLabel: lowcpu { cpus = 2 } + withLabel: midcpu { cpus = 4 } + withLabel: highcpu { cpus = 8 } +} +executor { + cpus = 8 +} +HERE + +# --------------------------------------------------------------------------- +# Step 1: process the full Wu 2021 breast-cancer scRNA-seq atlas +# --------------------------------------------------------------------------- +cat > /tmp/params_wu_full.yaml << HERE +param_list: + - id: $full_id + dataset_name: Wu 2021 Human Breast Cancer scRNAseq + dataset_organism: Homo sapiens + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +nextflow run . \ + -main-script target/nextflow/datasets/workflows/process_wu_human_breast_cancer_sc/main.nf \ + -profile docker \ + -resume \ + -c /tmp/labels_8cpu.config \ + -params-file /tmp/params_wu_full.yaml + +# --------------------------------------------------------------------------- +# Step 2: subsample the processed atlas to a small test reference (~400 cells). +# --------------------------------------------------------------------------- +cat > /tmp/params_wu_subsample.yaml << HERE +param_list: + - id: $test_id + input: $publish_dir/$full_id/dataset.h5ad + +n_obs: 400 +n_vars: 10000 +seed: 0 +output: "\$id/dataset.h5ad" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +nextflow run . \ + -main-script target/nextflow/datasets/processors/subsample/main.nf \ + -profile docker \ + -resume \ + -c /tmp/labels_8cpu.config \ + -params-file /tmp/params_wu_subsample.yaml + +# sync the small test reference to s3 +aws s3 sync --profile op \ + "resources_test/common/$test_id" \ + "s3://openproblems-data/resources_test/common/$test_id" \ + --delete --dryrun diff --git a/scripts/create_test_resources/2021_wu_human_breast_cancer_scrnaseq_nebius.sh b/scripts/create_test_resources/2021_wu_human_breast_cancer_scrnaseq_nebius.sh new file mode 100644 index 000000000..2aec9c2cb --- /dev/null +++ b/scripts/create_test_resources/2021_wu_human_breast_cancer_scrnaseq_nebius.sh @@ -0,0 +1,83 @@ +#!/bin/bash + +# Get the root of the repository +REPO_ROOT=$(git rev-parse --show-toplevel) + +# Ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Matching single-cell reference for the atera human breast-cancer spatial test sample. +# Runs on Nebius via tw launch and publishes to scratch (no local run). +# +# The process_wu_human_breast_cancer_sc workflow does NOT subsample internally, so this +# is a two-step chain (as with the yao mouse-brain test SC, which only gets a small +# reference because its loader subsamples): +# 1. run the workflow -> full processed SC atlas on scratch +# 2. chain the openproblems `subsample` module -> ~400-cell test reference on scratch +publish_dir="/scratch/task_ist_preprocessing/resources_test/common" + +full_id="2021_wu_human_breast_cancer_scrnaseq_full" +test_id="2021_wu_human_breast_cancer_scrnaseq" + +# --------------------------------------------------------------------------- +# Step 1: process the full Wu 2021 breast-cancer scRNA-seq atlas +# --------------------------------------------------------------------------- +cat > /tmp/params_wu_full.yaml << HERE +param_list: + - id: $full_id + dataset_name: Wu 2021 Human Breast Cancer scRNAseq + dataset_organism: Homo sapiens + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +#tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ +# --revision build/main \ +# --pull-latest \ +# --main-script target/nextflow/datasets/workflows/process_wu_human_breast_cancer_sc/main.nf \ +# --workspace 167877437119966 \ +# --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ +# --params-file /tmp/params_wu_full.yaml \ +# --config src/base/labels_nebius.config \ +# --labels test_resources,wu_human_breast_cancer_sc \ +# --wait SUCCEEDED + +# --------------------------------------------------------------------------- +# Step 2: subsample the processed atlas to a small test reference (~400 cells). +# Requires step 1 to have finished (see --wait SUCCEEDED above); if your tw CLI +# predates --wait, drop it and launch this step manually once step 1 completes. +# --------------------------------------------------------------------------- +cat > /tmp/params_wu_subsample.yaml << HERE +param_list: + - id: $test_id + input: $publish_dir/$full_id/dataset.h5ad + +n_obs: 400 +n_vars: 10000 +seed: 0 +output: "\$id/dataset.h5ad" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/processors/subsample/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_wu_subsample.yaml \ + --config src/base/labels_nebius.config \ + --labels test_resources,subsample + +# The small test reference lands under $publish_dir/$test_id/dataset.h5ad. +# To publish it as a test resource afterwards, sync it from scratch to S3, e.g.: +# aws s3 sync --profile op \ +# "$publish_dir/$test_id" \ +# "s3://openproblems-data/resources_test/common/$test_id" \ +# --delete --dryrun diff --git a/scripts/create_test_resources/2023_10x_mouse_brain_xenium_rep1_nebius.sh b/scripts/create_test_resources/2023_10x_mouse_brain_xenium_rep1_nebius.sh new file mode 100644 index 000000000..a6fea0017 --- /dev/null +++ b/scripts/create_test_resources/2023_10x_mouse_brain_xenium_rep1_nebius.sh @@ -0,0 +1,53 @@ +#!/bin/bash + +# Get the root of the repository +REPO_ROOT=$(git rev-parse --show-toplevel) + +# Ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Nebius/Seqera (tw launch) variant of 2023_10x_mouse_brain_xenium_rep1.sh. +# Publishes the cropped test resource to scratch instead of the local resources_test/. +# The tenx_xenium loader accepts a directory, a zip, or a download url and stages/extracts +# it itself, so the 10x zip URL is passed straight through (no local wget/unzip step). +publish_dir="/scratch/task_ist_preprocessing/resources_test/common" + +cat > /tmp/params_mouse_brain_xenium_test.yaml << HERE +param_list: + - id: 2023_10x_mouse_brain_xenium_rep1 + input: https://cf.10xgenomics.com/samples/xenium/1.0.2/Xenium_V1_FF_Mouse_Brain_MultiSection_1/Xenium_V1_FF_Mouse_Brain_MultiSection_1_outs.zip + segmentation_id: [cell, nucleus] + dataset_name: Xenium V1 Fresh Frozen Mouse Brain rep1 + dataset_url: https://www.10xgenomics.com/datasets/fresh-frozen-mouse-brain-replicates-1-standard + dataset_summary: Demonstration of gene expression profiling for fresh frozen mouse brain on the Xenium platform. + dataset_description: Demonstration of gene expression profiling for fresh frozen mouse brain on the Xenium platform using the pre-designed Mouse Brain Gene Expression Panel (v1). + dataset_organism: mus_musculus + dataset_reference: NA + crop_region_min_x: 10000 + crop_region_max_x: 11000 + crop_region_min_y: 10000 + crop_region_max_y: 11000 + +output_dataset: "\$id/dataset.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_tenx_xenium/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_mouse_brain_xenium_test.yaml \ + --config src/base/labels_nebius.config \ + --labels test_resources,tenx_xenium + +# The cropped dataset lands under $publish_dir/2023_10x_mouse_brain_xenium_rep1/. +# To publish it as a test resource afterwards, sync it from scratch to S3, e.g.: +# aws s3 sync --profile op \ +# "$publish_dir/2023_10x_mouse_brain_xenium_rep1" \ +# "s3://openproblems-data/resources_test/common/2023_10x_mouse_brain_xenium_rep1" \ +# --delete --dryrun diff --git a/scripts/create_test_resources/2023_yao_mouse_brain_scrnaseq_10xv2_nebius.sh b/scripts/create_test_resources/2023_yao_mouse_brain_scrnaseq_10xv2_nebius.sh new file mode 100644 index 000000000..bf036f346 --- /dev/null +++ b/scripts/create_test_resources/2023_yao_mouse_brain_scrnaseq_10xv2_nebius.sh @@ -0,0 +1,59 @@ +#!/bin/bash + +# Get the root of the repository +REPO_ROOT=$(git rev-parse --show-toplevel) + +# Ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Nebius/Seqera (tw launch) variant of 2023_yao_mouse_brain_scrnaseq_10xv2.sh. +# Matching single-cell reference for the 10x Xenium mouse-brain spatial test sample. +# Publishes to scratch instead of the local resources_test/. +# +# The allen_brain_cell_atlas loader downloads the requested regions from the ABCA directly +# (no local input), and subsamples during loading. This workflow does NOT use the +# processors/subsample component, so it is a single step (unlike the Wu/Zuani SC chains). +# +# NOTE: the current workflow subsamples via the loader's --sample_n_obs (cells only); the +# older do_subsample/n_obs/n_vars params in the local script are no longer part of the +# workflow interface and are silently ignored by findStates (which would yield the FULL +# OLF+TH regions rather than a small test reference). Use sample_n_obs here. +publish_dir="/scratch/task_ist_preprocessing/resources_test/common" + +cat > /tmp/params_yao_mouse_brain_test.yaml << HERE +param_list: + - id: 2023_yao_mouse_brain_scrnaseq_10xv2 + regions: [OLF, TH] + dataset_name: ABCA Mouse Brain scRNAseq + dataset_url: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE246717 + dataset_reference: 10.1038/s41586-023-06812-z + dataset_summary: A high-resolution scRNAseq atlas of cell types in the whole mouse brain + dataset_description: See dataset_reference for more information. Note that we only took the 10xv2 data from the dataset. + dataset_organism: mus_musculus + sample_n_obs: 400 + sample_seed: 0 + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_allen_brain_cell_atlas/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_yao_mouse_brain_test.yaml \ + --config src/base/labels_nebius.config \ + --labels test_resources,allen_brain_cell_atlas + +# The subsampled reference lands under $publish_dir/2023_yao_mouse_brain_scrnaseq_10xv2/. +# To publish it as a test resource afterwards, sync it from scratch to S3, e.g.: +# aws s3 sync --profile op \ +# "$publish_dir/2023_yao_mouse_brain_scrnaseq_10xv2" \ +# "s3://openproblems-data/resources_test/common/2023_yao_mouse_brain_scrnaseq_10xv2" \ +# --delete --dryrun diff --git a/scripts/create_test_resources/2024_zuani_human_nsclc_scrnaseq.sh b/scripts/create_test_resources/2024_zuani_human_nsclc_scrnaseq.sh new file mode 100644 index 000000000..d27e40155 --- /dev/null +++ b/scripts/create_test_resources/2024_zuani_human_nsclc_scrnaseq.sh @@ -0,0 +1,90 @@ +#!/bin/bash + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Local (nextflow run) variant of 2024_zuani_human_nsclc_scrnaseq_nebius.sh. +# Matching single-cell reference for the CosMx human lung-cancer (NSCLC) spatial test sample. +# Publishes to the local resources_test/ instead of scratch. +# +# The process_zuani_human_nsclc_sc workflow does NOT subsample internally, so this is a +# two-step chain (as in the nebius script): +# 1. run the workflow -> full processed NSCLC atlas under resources_test/common (~900k cells) +# 2. chain the openproblems `subsample` module -> ~400-cell test reference +# `nextflow run` is synchronous, so step 2 runs only after step 1 finishes (no --wait needed). +# The zuani loader has its input FTP URL hardcoded, so no --input is passed. +# NOTE: step 1 processes the full ~900k-cell atlas locally and is memory/time heavy. +publish_dir="resources_test/common" + +full_id="2024_zuani_human_nsclc_scrnaseq_full" +test_id="2024_zuani_human_nsclc_scrnaseq" + +# Cap per-process CPU requests to the 8 cores available on this machine. Component +# labels otherwise request up to highcpu=30, which exceeds the machine and the +# Nextflow local executor would then fail to schedule those processes. +cat > /tmp/labels_8cpu.config << 'HERE' +process { + withLabel: lowcpu { cpus = 2 } + withLabel: midcpu { cpus = 4 } + withLabel: highcpu { cpus = 8 } +} +executor { + cpus = 8 +} +HERE + +# --------------------------------------------------------------------------- +# Step 1: process the full Zuani 2024 NSCLC scRNA-seq atlas +# --------------------------------------------------------------------------- +cat > /tmp/params_zuani_full.yaml << HERE +param_list: + - id: $full_id + dataset_name: Zuani 2024 Human NSCLC scRNAseq + dataset_organism: human + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +nextflow run . \ + -main-script target/nextflow/datasets/workflows/process_zuani_human_nsclc_sc/main.nf \ + -profile docker \ + -resume \ + -c /tmp/labels_8cpu.config \ + -params-file /tmp/params_zuani_full.yaml + +# --------------------------------------------------------------------------- +# Step 2: subsample the processed atlas to a small test reference (~400 cells). +# --------------------------------------------------------------------------- +cat > /tmp/params_zuani_subsample.yaml << HERE +param_list: + - id: $test_id + input: $publish_dir/$full_id/dataset.h5ad + +n_obs: 400 +n_vars: 10000 +seed: 0 +output: "\$id/dataset.h5ad" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +nextflow run . \ + -main-script target/nextflow/datasets/processors/subsample/main.nf \ + -profile docker \ + -resume \ + -c /tmp/labels_8cpu.config \ + -params-file /tmp/params_zuani_subsample.yaml + +# sync the small test reference to s3 +aws s3 sync --profile op \ + "resources_test/common/$test_id" \ + "s3://openproblems-data/resources_test/common/$test_id" \ + --delete --dryrun diff --git a/scripts/create_test_resources/2024_zuani_human_nsclc_scrnaseq_nebius.sh b/scripts/create_test_resources/2024_zuani_human_nsclc_scrnaseq_nebius.sh new file mode 100644 index 000000000..a3339ea05 --- /dev/null +++ b/scripts/create_test_resources/2024_zuani_human_nsclc_scrnaseq_nebius.sh @@ -0,0 +1,83 @@ +#!/bin/bash + +# Get the root of the repository +REPO_ROOT=$(git rev-parse --show-toplevel) + +# Ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Matching single-cell reference for the CosMx human lung-cancer (NSCLC) spatial test +# sample. Runs on Nebius via tw launch and publishes to scratch (no local run). +# +# The process_zuani_human_nsclc_sc workflow does NOT subsample internally, so this is a +# two-step chain: +# 1. run the workflow -> full processed NSCLC atlas on scratch (~900k cells) +# 2. chain the openproblems `subsample` module -> ~400-cell test reference on scratch +# The zuani loader has its input FTP URL hardcoded, so no --input is passed. +publish_dir="/scratch/task_ist_preprocessing/resources_test/common" + +full_id="2024_zuani_human_nsclc_scrnaseq_full" +test_id="2024_zuani_human_nsclc_scrnaseq" + +# --------------------------------------------------------------------------- +# Step 1: process the full Zuani 2024 NSCLC scRNA-seq atlas +# --------------------------------------------------------------------------- +cat > /tmp/params_zuani_full.yaml << HERE +param_list: + - id: $full_id + dataset_name: Zuani 2024 Human NSCLC scRNAseq + dataset_organism: human + +output_dataset: "\$id/dataset.h5ad" +output_meta: "\$id/dataset_meta.yaml" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +#tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ +# --revision build/main \ +# --pull-latest \ +# --main-script target/nextflow/datasets/workflows/process_zuani_human_nsclc_sc/main.nf \ +# --workspace 167877437119966 \ +# --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ +# --params-file /tmp/params_zuani_full.yaml \ +# --config src/base/labels_nebius.config \ +# --labels test_resources,zuani_human_nsclc_sc \ +# --wait SUCCEEDED + +# --------------------------------------------------------------------------- +# Step 2: subsample the processed atlas to a small test reference (~400 cells). +# Requires step 1 to have finished (see --wait SUCCEEDED above); if your tw CLI +# predates --wait, drop it and launch this step manually once step 1 completes. +# --------------------------------------------------------------------------- +cat > /tmp/params_zuani_subsample.yaml << HERE +param_list: + - id: $test_id + input: $publish_dir/$full_id/dataset.h5ad + +n_obs: 400 +n_vars: 10000 +seed: 0 +output: "\$id/dataset.h5ad" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/processors/subsample/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_zuani_subsample.yaml \ + --config src/base/labels_nebius.config \ + --labels test_resources,subsample + +# The small test reference lands under $publish_dir/$test_id/dataset.h5ad. +# To publish it as a test resource afterwards, sync it from scratch to S3, e.g.: +# aws s3 sync --profile op \ +# "$publish_dir/$test_id" \ +# "s3://openproblems-data/resources_test/common/$test_id" \ +# --delete --dryrun diff --git a/scripts/create_test_resources/2026_10x_human_breast_cancer_atera.sh b/scripts/create_test_resources/2026_10x_human_breast_cancer_atera.sh new file mode 100644 index 000000000..d4e2a51b1 --- /dev/null +++ b/scripts/create_test_resources/2026_10x_human_breast_cancer_atera.sh @@ -0,0 +1,57 @@ +#!/bin/bash + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +if [ ! -d temp/datasets/10x_atera/2026_10x_human_breast_cancer_atera ]; then + mkdir -p temp/datasets/10x_atera/2026_10x_human_breast_cancer_atera +fi +if [ ! -f temp/datasets/10x_atera/2026_10x_human_breast_cancer_atera/WTA_Preview_FFPE_Breast_Cancer_xe_outs.zip ]; then + wget -O temp/datasets/10x_atera/2026_10x_human_breast_cancer_atera/WTA_Preview_FFPE_Breast_Cancer_xe_outs.zip \ + https://s3-us-west-2.amazonaws.com/10x.files/samples/atera/dev/WTA_Preview_FFPE_Breast_Cancer/WTA_Preview_FFPE_Breast_Cancer_outs.zip +fi + +cat > /tmp/params.yaml << HERE +param_list: + - id: 2026_10x_human_breast_cancer_atera + input: temp/datasets/10x_atera/2026_10x_human_breast_cancer_atera/WTA_Preview_FFPE_Breast_Cancer_xe_outs.zip + segmentation_id: + - cell + - nucleus + dataset_name: Atera WTA FFPE Human Breast Cancer + dataset_url: https://www.10xgenomics.com/datasets/atera-wta-ffpe-human-breast-cancer + dataset_summary: Preview dataset showcasing the pre-commercial Atera Whole Transcriptome Assay (WTA) applied to FFPE human breast cancer tissue. + dataset_description: "This human FFPE breast cancer data showcases results using the pre-commercial version of the Atera Whole Transcriptome Assay (WTA). A single 5µm FFPE section of breast cancer tissue (DCIS Grade 3, T1c N0 M0) was analyzed, yielding 170,057 detected cells with a median of 2,116 transcripts per cell." + dataset_organism: homo_sapiens + dataset_reference: NA + # crop_region_* are in the "global" coordinate system, which for Xenium/Atera + # is IMAGE PIXELS (0.2125 um/px), not microns: a 1000-unit box is ~212.5 um. + # This window (px) recenters onto dense tumor tissue (~572 cells); the old + # 5000-6000 box landed on a near-empty corner of the section (~13 cells). + crop_region_min_x: 17000 + crop_region_max_x: 18000 + crop_region_min_y: 18000 + crop_region_max_y: 19000 + +publish_dir: resources_test/common +output_dataset: '\$id/dataset.zarr' +output_state: '\$id/state.yaml' +HERE + +# convert to zarr +nextflow run . \ + -main-script target/nextflow/datasets/workflows/process_tenx_atera/main.nf \ + -profile docker \ + -resume \ + -params-file /tmp/params.yaml + +# sync to s3 +aws s3 sync --profile op \ + "resources_test/common/2026_10x_human_breast_cancer_atera" \ + "s3://openproblems-data/resources_test/common/2026_10x_human_breast_cancer_atera" \ + --delete --dryrun \ No newline at end of file diff --git a/scripts/create_test_resources/2026_10x_human_breast_cancer_atera_nebius.sh b/scripts/create_test_resources/2026_10x_human_breast_cancer_atera_nebius.sh new file mode 100644 index 000000000..a30155135 --- /dev/null +++ b/scripts/create_test_resources/2026_10x_human_breast_cancer_atera_nebius.sh @@ -0,0 +1,56 @@ +#!/bin/bash + +# Get the root of the repository +REPO_ROOT=$(git rev-parse --show-toplevel) + +# Ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Nebius/Seqera (tw launch) variant of 2026_10x_human_breast_cancer_atera.sh. +# Publishes the cropped test resource to scratch instead of the local resources_test/. +# The atera zip URL is passed straight through (Nextflow stages it), so no wget step. +publish_dir="/scratch/task_ist_preprocessing/resources_test/common" + +cat > /tmp/params_atera_test.yaml << HERE +param_list: + - id: 2026_10x_human_breast_cancer_atera + input: https://s3-us-west-2.amazonaws.com/10x.files/samples/atera/dev/WTA_Preview_FFPE_Breast_Cancer/WTA_Preview_FFPE_Breast_Cancer_outs.zip + segmentation_id: [cell, nucleus] + dataset_name: Atera WTA FFPE Human Breast Cancer + dataset_url: https://www.10xgenomics.com/datasets/atera-wta-ffpe-human-breast-cancer + dataset_summary: Preview dataset showcasing the pre-commercial Atera Whole Transcriptome Assay (WTA) applied to FFPE human breast cancer tissue. + dataset_description: "This human FFPE breast cancer data showcases results using the pre-commercial version of the Atera Whole Transcriptome Assay (WTA). A single 5µm FFPE section of breast cancer tissue (DCIS Grade 3, T1c N0 M0) was analyzed, yielding 170,057 detected cells with a median of 2,116 transcripts per cell." + dataset_organism: homo_sapiens + dataset_reference: NA + # crop_region_* are in the "global" coordinate system, which for Xenium/Atera + # is IMAGE PIXELS (0.2125 um/px), not microns: a 1000-unit box is ~212.5 um. + # This window (px) recenters onto dense tumor tissue (~572 cells); the old + # 5000-6000 box landed on a near-empty corner of the section (~13 cells). + crop_region_min_x: 17000 + crop_region_max_x: 18000 + crop_region_min_y: 18000 + crop_region_max_y: 19000 + +output_dataset: "\$id/dataset.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_tenx_atera/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_atera_test.yaml \ + --config src/base/labels_nebius.config \ + --labels test_resources,atera + +# The cropped dataset lands under $publish_dir/2026_10x_human_breast_cancer_atera/. +# To publish it as a test resource afterwards, sync it from scratch to S3, e.g.: +# aws s3 sync --profile op \ +# "$publish_dir/2026_10x_human_breast_cancer_atera" \ +# "s3://openproblems-data/resources_test/common/2026_10x_human_breast_cancer_atera" \ +# --delete --dryrun diff --git a/scripts/create_test_resources/2026_bruker_human_lung_cancer_cosmx.sh b/scripts/create_test_resources/2026_bruker_human_lung_cancer_cosmx.sh new file mode 100644 index 000000000..900238d87 --- /dev/null +++ b/scripts/create_test_resources/2026_bruker_human_lung_cancer_cosmx.sh @@ -0,0 +1,53 @@ +#!/bin/bash + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Smallest of the CosMx archives mirrored to S3 (Lung9 Rep1: ~1.0 GiB flat + ~23.3 GiB +# morphology). The bruker_cosmx_nsclc loader streams these straight from S3 and only +# extracts what sopa needs, so no local download step is required (input_raw / +# input_morphology are plain strings, not staged files). +raw_dir="s3://openproblems-data/resources/raw_data/bruker_cosmx" + +cat > /tmp/params.yaml << HERE +param_list: + - id: 2026_bruker_human_lung_cancer_cosmx + input_raw: $raw_dir/Lung9_Rep1+SMI+Flat+data.tar.gz + input_morphology: $raw_dir/Lung9_Rep1+RawMorphologyImages.tar.gz + segmentation_id: + - cell + dataset_name: Bruker CosMx Human Lung Cancer Lung9 Rep1 + dataset_url: https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/ + dataset_summary: Bruker CosMx Human Lung Cancer Lung9 Rep1 dataset on FFPE. + dataset_description: "Bruker CosMx Human Lung Cancer Lung9 Rep1 dataset on FFPE. Adenocarcinoma, G3, T3N1M0, IIIA, 65% tumour content." + dataset_organism: homo_sapiens + dataset_reference: NA + # NOTE: coordinates are in the dataset's global coordinate system. Adjust the box + # if it lands outside the tissue (an out-of-range crop yields an empty dataset). + crop_region_min_x: 5000 + crop_region_max_x: 6000 + crop_region_min_y: 5000 + crop_region_max_y: 6000 + +publish_dir: resources_test/common +output_dataset: '\$id/dataset.zarr' +output_state: '\$id/state.yaml' +HERE + +# convert to zarr +nextflow run . \ + -main-script target/nextflow/datasets/workflows/process_bruker_cosmx_nsclc/main.nf \ + -profile docker \ + -resume \ + -params-file /tmp/params.yaml + +# sync to s3 +aws s3 sync --profile op \ + "resources_test/common/2026_bruker_human_lung_cancer_cosmx" \ + "s3://openproblems-data/resources_test/common/2026_bruker_human_lung_cancer_cosmx" \ + --delete --dryrun diff --git a/scripts/create_test_resources/2026_bruker_human_lung_cancer_cosmx_nebius.sh b/scripts/create_test_resources/2026_bruker_human_lung_cancer_cosmx_nebius.sh new file mode 100644 index 000000000..900350987 --- /dev/null +++ b/scripts/create_test_resources/2026_bruker_human_lung_cancer_cosmx_nebius.sh @@ -0,0 +1,57 @@ +#!/bin/bash + +# Get the root of the repository +REPO_ROOT=$(git rev-parse --show-toplevel) + +# Ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Nebius/Seqera (tw launch) variant of 2026_bruker_human_lung_cancer_cosmx.sh. +# Publishes the cropped test resource to scratch instead of the local resources_test/. +# Smallest CosMx archive pair (Lung9 Rep1); the bruker_cosmx_nsclc loader streams both +# straight from S3 and extracts only what sopa needs. +publish_dir="/scratch/task_ist_preprocessing/resources_test/common" +raw_dir="s3://openproblems-data/resources/raw_data/bruker_cosmx" + +cat > /tmp/params_cosmx_test.yaml << HERE +param_list: + - id: 2026_bruker_human_lung_cancer_cosmx + input_raw: $raw_dir/Lung9_Rep1+SMI+Flat+data.tar.gz + input_morphology: $raw_dir/Lung9_Rep1+RawMorphologyImages.tar.gz + segmentation_id: ["cell"] + dataset_name: Bruker CosMx Human Lung Cancer Lung9 Rep1 + dataset_url: https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/ + dataset_summary: Bruker CosMx Human Lung Cancer Lung9 Rep1 dataset on FFPE. + dataset_description: "Bruker CosMx Human Lung Cancer Lung9 Rep1 dataset on FFPE. Adenocarcinoma, G3, T3N1M0, IIIA, 65% tumour content." + dataset_organism: homo_sapiens + dataset_reference: NA + # NOTE: coordinates are in the dataset's global coordinate system. Adjust the box + # if it lands outside the tissue (an out-of-range crop yields an empty dataset). + crop_region_min_x: 5000 + crop_region_max_x: 6000 + crop_region_min_y: 5000 + crop_region_max_y: 6000 + +output_dataset: "\$id/dataset.zarr" +output_state: "\$id/state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/datasets/workflows/process_bruker_cosmx_nsclc/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_cosmx_test.yaml \ + --config src/base/labels_nebius.config \ + --labels test_resources,bruker_cosmx_nsclc + +# The cropped dataset lands under $publish_dir/2026_bruker_human_lung_cancer_cosmx/. +# To publish it as a test resource afterwards, sync it from scratch to S3, e.g.: +# aws s3 sync --profile op \ +# "$publish_dir/2026_bruker_human_lung_cancer_cosmx" \ +# "s3://openproblems-data/resources_test/common/2026_bruker_human_lung_cancer_cosmx" \ +# --delete --dryrun diff --git a/scripts/create_test_resources/process_dataset_atera_test.sh b/scripts/create_test_resources/process_dataset_atera_test.sh new file mode 100644 index 000000000..bd2f41a7e --- /dev/null +++ b/scripts/create_test_resources/process_dataset_atera_test.sh @@ -0,0 +1,46 @@ +#!/bin/bash + +# Build the combined atera *test* dataset used by results/Dataset_report.qmd. +# +# Why this exists: the loaders emit the image element as `morphology_mip`; only +# process_dataset renames it to the API-required `image` (file_common_ist.yaml). +# The resources_test loader outputs (resources_test/common//dataset.zarr) are +# raw loader outputs, so they keep `morphology_mip` and lack a panel-subset SC — +# the report can't read them directly. Running process_dataset here is the +# workaround: it renames morphology_mip -> image, subsets the SC to the shared +# panel, and writes a combined dataset (raw_ist.zarr + scrnaseq_reference.h5ad), +# exactly like test_pipeline.sh does for mouse_brain_combined. +# +# The SC input is the subsampled Wu test reference (tw run 4DcTrWjb5pkW2u). + +set -e + +REPO_ROOT=$(git rev-parse --show-toplevel) +cd "$REPO_ROOT" + +SP=resources_test/common/2026_10x_human_breast_cancer_atera/dataset.zarr +SC=resources_test/common/2021_wu_human_breast_cancer_scrnaseq/dataset.h5ad +OUT_DIR=resources_test/task_ist_preprocessing/atera_breast_cancer_test_combined + +rm -rf "$OUT_DIR" +mkdir -p "$OUT_DIR" + +viash run src/data_processors/process_dataset/config.vsh.yaml -- \ + --dataset_id atera_breast_cancer_test_combined \ + --dataset_name "Test atera WTA human breast cancer combined 2021 Wu scRNAseq" \ + --dataset_url "" \ + --dataset_reference "" \ + --dataset_summary "Test crop of Atera WTA FFPE human breast cancer + subsampled Wu 2021 breast scRNAseq" \ + --dataset_description "Cropped test resource (Atera WTA FFPE human breast cancer) combined with a subsampled Wu 2021 breast scRNAseq reference, for Dataset_report validation." \ + --dataset_organism "homo_sapiens" \ + --input_sc "$SC" \ + --input_sp "$SP" \ + --output_sc "$OUT_DIR/scrnaseq_reference.h5ad" \ + --output_sp "$OUT_DIR/raw_ist.zarr" + +# The combined test dataset lands under $OUT_DIR (raw_ist.zarr has element "image"). +# To publish it as a test resource, sync it to S3, e.g.: +# aws s3 sync --profile op \ +# "$OUT_DIR" \ +# "s3://openproblems-data/resources_test/task_ist_preprocessing/atera_breast_cancer_test_combined" \ +# --delete --dryrun diff --git a/scripts/create_test_resources/process_dataset_bruker_cosmx_test.sh b/scripts/create_test_resources/process_dataset_bruker_cosmx_test.sh new file mode 100644 index 000000000..f3fb38d57 --- /dev/null +++ b/scripts/create_test_resources/process_dataset_bruker_cosmx_test.sh @@ -0,0 +1,48 @@ +#!/bin/bash + +# Build the combined Bruker CosMx lung-cancer *test* dataset used by +# results/Dataset_report.qmd. +# +# Why this exists: the loaders emit the image element as `morphology_mip`; only +# process_dataset renames it to the API-required `image` (file_common_ist.yaml). +# The resources_test loader outputs (resources_test/common//dataset.zarr) are +# raw loader outputs, so they keep `morphology_mip` and lack a panel-subset SC — +# the report can't read them directly. Running process_dataset here is the +# workaround: it renames morphology_mip -> image, subsets the SC to the shared +# panel, and writes a combined dataset (raw_ist.zarr + scrnaseq_reference.h5ad), +# exactly like test_pipeline.sh does for mouse_brain_combined. +# +# The SC input is the subsampled Zuani NSCLC test reference +# (scripts/create_test_resources/2024_zuani_human_nsclc_scrnaseq_nebius.sh). + +set -e + +REPO_ROOT=$(git rev-parse --show-toplevel) +cd "$REPO_ROOT" + +SP=resources_test/common/2026_bruker_human_lung_cancer_cosmx/dataset.zarr +SC=resources_test/common/2024_zuani_human_nsclc_scrnaseq/dataset.h5ad +OUT_DIR=resources_test/task_ist_preprocessing/bruker_lung_cancer_cosmx_test_combined + +rm -rf "$OUT_DIR" +mkdir -p "$OUT_DIR" + +viash run src/data_processors/process_dataset/config.vsh.yaml -- \ + --dataset_id bruker_lung_cancer_cosmx_test_combined \ + --dataset_name "Test human lung cancer combined Bruker CosMx Lung9 rep1 2024 Zuani scRNAseq" \ + --dataset_url "https://nanostring.com/products/cosmx-spatial-molecular-imager/ffpe-dataset/nsclc-ffpe-dataset/" \ + --dataset_reference "" \ + --dataset_summary "Test crop of Bruker CosMx human lung cancer (Lung9 rep1) + subsampled 2024 Zuani NSCLC scRNAseq" \ + --dataset_description "Cropped test resource (Bruker CosMx human lung cancer, Lung9 rep1) combined with a subsampled Zuani 2024 NSCLC scRNAseq reference, for Dataset_report validation." \ + --dataset_organism "homo_sapiens" \ + --input_sc "$SC" \ + --input_sp "$SP" \ + --output_sc "$OUT_DIR/scrnaseq_reference.h5ad" \ + --output_sp "$OUT_DIR/raw_ist.zarr" + +# The combined test dataset lands under $OUT_DIR (raw_ist.zarr has element "image"). +# To publish it as a test resource, sync it to S3, e.g.: +# aws s3 sync --profile op \ +# "$OUT_DIR" \ +# "s3://openproblems-data/resources_test/task_ist_preprocessing/bruker_lung_cancer_cosmx_test_combined" \ +# --delete --dryrun diff --git a/scripts/create_test_resources/process_dataset_mouse_brain_xenium_test.sh b/scripts/create_test_resources/process_dataset_mouse_brain_xenium_test.sh new file mode 100644 index 000000000..074a61213 --- /dev/null +++ b/scripts/create_test_resources/process_dataset_mouse_brain_xenium_test.sh @@ -0,0 +1,47 @@ +#!/bin/bash + +# Build the combined mouse-brain Xenium *test* dataset used by results/Dataset_report.qmd. +# +# Why this exists: the loaders emit the image element as `morphology_mip`; only +# process_dataset renames it to the API-required `image` (file_common_ist.yaml). +# The resources_test loader outputs (resources_test/common//dataset.zarr) are +# raw loader outputs, so they keep `morphology_mip` and lack a panel-subset SC — +# the report can't read them directly. Running process_dataset here is the +# workaround: it renames morphology_mip -> image, subsets the SC to the shared +# panel, and writes a combined dataset (raw_ist.zarr + scrnaseq_reference.h5ad), +# exactly like test_pipeline.sh does for mouse_brain_combined. +# +# The SC input is the subsampled Yao mouse-brain test reference +# (scripts/create_test_resources/2023_yao_mouse_brain_scrnaseq_10xv2_nebius.sh). + +set -e + +REPO_ROOT=$(git rev-parse --show-toplevel) +cd "$REPO_ROOT" + +SP=resources_test/common/2023_10x_mouse_brain_xenium_rep1/dataset.zarr +SC=resources_test/common/2023_yao_mouse_brain_scrnaseq_10xv2/dataset.h5ad +OUT_DIR=resources_test/task_ist_preprocessing/mouse_brain_xenium_test_combined + +rm -rf "$OUT_DIR" +mkdir -p "$OUT_DIR" + +viash run src/data_processors/process_dataset/config.vsh.yaml -- \ + --dataset_id mouse_brain_xenium_test_combined \ + --dataset_name "Test mouse brain combined 2023 10x Xenium rep1 2023 Yao scRNAseq" \ + --dataset_url "https://www.10xgenomics.com/datasets/fresh-frozen-mouse-brain-replicates-1-standard" \ + --dataset_reference "" \ + --dataset_summary "Test crop of 10x Xenium fresh-frozen mouse brain (rep1) + subsampled 2023 Yao mouse brain scRNAseq" \ + --dataset_description "Cropped test resource (10x Xenium fresh-frozen mouse brain, rep1) combined with a subsampled Yao 2023 mouse brain scRNAseq reference, for Dataset_report validation." \ + --dataset_organism "mus_musculus" \ + --input_sc "$SC" \ + --input_sp "$SP" \ + --output_sc "$OUT_DIR/scrnaseq_reference.h5ad" \ + --output_sp "$OUT_DIR/raw_ist.zarr" + +# The combined test dataset lands under $OUT_DIR (raw_ist.zarr has element "image"). +# To publish it as a test resource, sync it to S3, e.g.: +# aws s3 sync --profile op \ +# "$OUT_DIR" \ +# "s3://openproblems-data/resources_test/task_ist_preprocessing/mouse_brain_xenium_test_combined" \ +# --delete --dryrun diff --git a/scripts/run_benchmark/param_sweep/mapmycells_params.yaml b/scripts/run_benchmark/param_sweep/mapmycells_params.yaml new file mode 100644 index 000000000..92400a3e9 --- /dev/null +++ b/scripts/run_benchmark/param_sweep/mapmycells_params.yaml @@ -0,0 +1,34 @@ +# Parameter sweep for the mapmycells (Allen cell_type_mapper / MapMyCells) cell-type +# annotation method. Committed source of truth for run_test_mapmycells_nebius.sh (read +# from GitHub via a raw URL, since the Nebius compute env pulls the repo but cannot see +# the launch host's local files). +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf). For every method the workflow builds: +# * one "default" variant using the `default:` args below, and +# * one extra variant per value in each `sweep:` list, with that ONE arg overridden. +# The benchmark varies a SINGLE parameter at a time (a "star" around the default, not +# a full grid), so total mapmycells variants = 1 default + sum(sweep list lengths) = 5. +# +# See src/methods_cell_type_annotation/mapmycells/NOTES.md ("Optimization / tuning") for +# the rationale. +# +# ⚠️ bootstrap_iteration / bootstrap_factor were JUST ADDED to config.vsh.yaml, so the +# build/main container must be rebuilt + pushed before this sweep runs (check-component). +parameters: + mapmycells: + # Baseline (fast anchor). NB the SHIPPED component default is (1, 1.0) = no + # bootstrapping. Here bootstrap_factor is pinned to the tool default 0.9 (not 1.0) + # ON PURPOSE: with factor 1.0 every bootstrap iteration is identical, which would + # make the bootstrap_iteration sweep below inert. 0.9 makes the axis meaningful. + default: + bootstrap_iteration: 1 + bootstrap_factor: 0.9 + sweep: + # bootstrap_iteration: how many times each query cell is re-mapped on a random + # 90% (bootstrap_factor) subset of markers before the majority vote. 1 (the + # default variant above) = single pass; the Allen tool default for bootstrapped + # correlation mapping is 100. Walks the shipped speed-tuned value up toward the + # tool default. 1 is omitted here (covered by the default variant); 100 is the + # slowest / most-robust endpoint. + bootstrap_iteration: [10, 25, 50, 100] diff --git a/scripts/run_benchmark/param_sweep/run_test_mapmycells_nebius.sh b/scripts/run_benchmark/param_sweep/run_test_mapmycells_nebius.sh new file mode 100644 index 000000000..86a20c7df --- /dev/null +++ b/scripts/run_benchmark/param_sweep/run_test_mapmycells_nebius.sh @@ -0,0 +1,109 @@ +#!/bin/bash + +# Nebius test run: all default methods + MapMyCells (mapmycells) cell-type annotation, +# with a parameter sweep over the Allen cell_type_mapper bootstrap knob +# (bootstrap_iteration). See src/methods_cell_type_annotation/mapmycells/NOTES.md +# ("Optimization / tuning"). +# +# mapmycells is a CPU-only method (Allen cell_type_mapper correlation mapping, no GPU), +# so it runs on the standard (non-GPU) compute env with no `gpu` label. +# +# ⚠️ NOT SUBMITTABLE AS-IS. bootstrap_iteration / bootstrap_factor were JUST ADDED to +# src/methods_cell_type_annotation/mapmycells/config.vsh.yaml, so the build/main +# container run by --revision below does NOT yet understand them. Before launching: +# 1. viash ns build --setup cachedbuild -q mapmycells (regenerate + rebuild image) +# 2. commit + push the config/script/params to build/main (and this params file to +# $params_branch) -- see the `check-component` skill to confirm the ghcr image +# is fresh (build_main tag) before the run. +# +# PARAMS-FILE CAVEAT (why the sweep is not an inline heredoc): +# `tw launch --params-file` is read client-side, but `method_parameters_yaml` +# is a path the WORKFLOW opens at runtime on the cloud (readYaml -> Nextflow +# file()). A local /tmp path does not exist there, and /scratch (where results +# publish) is READ-ONLY from the launch host -- which is why the binning +# method_params block is commented out in run_test_nebius.sh. file() does stage +# http(s):// though, and this repo is public, so we keep the sweep in a COMMITTED +# file (scripts/run_benchmark/param_sweep/mapmycells_params.yaml) and read it from +# GitHub via its raw URL. => the params file must be committed AND PUSHED to +# $params_branch before launching (edit the file there, not here, to change the sweep). + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +resources_test_s3="/scratch/task_ist_preprocessing/resources_test/task_ist_preprocessing/" +# Results publish to /scratch -- created and written by the cloud compute env, so +# the launcher does NOT create it here (it is read-only from the launch host). +publish_dir="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_mapmycells" + +# The sweep lives in a committed file, read from GitHub at runtime. $params_branch +# defaults to the branch you are on; the file must be pushed there on GitHub. (This +# is independent of --revision below, which selects the pipeline CODE to run.) +params_repo="openproblems-bio/task_ist_preprocessing" +params_branch="$(git rev-parse --abbrev-ref HEAD)" +params_url="https://raw.githubusercontent.com/${params_repo}/${params_branch}/scripts/run_benchmark/param_sweep/mapmycells_params.yaml" + +cat > /tmp/params_settings.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco + - mapmycells +expression_correction_methods: + - no_correction +gene_efficiency_correction_methods: + - no_correction +method_parameters_yaml: $params_url +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params.yaml << HERE +input_states: $resources_test_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir" +HERE + +# Fail early with a clear message if the params file isn't reachable on GitHub yet. +if ! curl -fsSL -o /dev/null "$params_url"; then + echo "ERROR: params file not reachable at:" >&2 + echo " $params_url" >&2 + echo "Commit and push scripts/run_benchmark/param_sweep/mapmycells_params.yaml to '$params_branch' first." >&2 + exit 1 +fi + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params.yaml \ + --entry-name auto \ + --config src/base/labels_nebius_test.config \ + --labels task_ist_preprocessing,test,mapmycells diff --git a/scripts/run_benchmark/param_sweeps_full/baysor_params.yaml b/scripts/run_benchmark/param_sweeps_full/baysor_params.yaml new file mode 100644 index 000000000..662348fbe --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/baysor_params.yaml @@ -0,0 +1,32 @@ +# Parameter sweep for the baysor transcript assignment method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_transcript_assignment.qmd +# from the sweep results in ~/projects/txsim_results/param_sweep_tests/transcript_assignment/baysor +# +# Selection: the 3 parameter families that reach the Pareto front (max cells, +# max negative marker purity) in the most datasets, tie-broken by how many datasets +# they beat `default` in and then by Euclidean distance from `default`. Within each +# family: every value on the front in >=1 dataset, plus the highest-distance value, +# topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total baysor variants = 1 default + 6 swept values. +parameters: + baysor: + default: + force_2d: true + min_molecules_per_cell: 50 + scale: -1.0 + scale_std: "25%" + n_clusters: 4 + prior_segmentation_confidence: 0.8 + sweep: + # on front in 2/3 datasets, beats default in 3; picked: 0.2 (front + max distance), 0.5 (distance top-up) + prior_segmentation_confidence: [0.2, 0.5] + # on front in 1/3 datasets, beats default in 3; picked: 10 (front + max distance), 25 (distance top-up) + min_molecules_per_cell: [10, 25] + # on front in 0/3 datasets, beats default in 1; picked: 8 (max distance), 6 (distance top-up) + n_clusters: [6, 8] diff --git a/scripts/run_benchmark/param_sweeps_full/clustermap_params.yaml b/scripts/run_benchmark/param_sweeps_full/clustermap_params.yaml new file mode 100644 index 000000000..56cb34f8c --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/clustermap_params.yaml @@ -0,0 +1,38 @@ +# Parameter sweep for the clustermap transcript assignment method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_transcript_assignment.qmd +# from the sweep results in ~/projects/txsim_results/param_sweep_tests/transcript_assignment/clustermap +# +# Selection: the 3 parameter families that reach the Pareto front (max cells, +# max negative marker purity) in the most datasets, tie-broken by how many datasets +# they beat `default` in and then by Euclidean distance from `default`. Within each +# family: every value on the front in >=1 dataset, plus the highest-distance value, +# topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total clustermap variants = 1 default + 8 swept values. +parameters: + clustermap: + default: + window_size: 700 + xy_radius: 40 + z_radius: 0 + fast_preprocess: false + gauss_blur: true + sigma: 1.0 + pct_filter: 0.0 + LOF: false + contamination: 0 + min_spot_per_cell: 5 + dapi_grid_interval: 5 + cell_num_threshold: 0.1 + sweep: + # on front in 0/3 datasets, beats default in 2; picked: 60 (max distance), 30 (distance top-up), 20 (distance top-up) + xy_radius: [20, 30, 60] + # on front in 0/3 datasets, beats default in 2; picked: 0.01 (max distance), 0.05 (distance top-up), 0.2 (distance top-up) + cell_num_threshold: [0.01, 0.05, 0.2] + # on front in 0/3 datasets, beats default in 2; picked: 0.05 (max distance), 0.1 (distance top-up) + pct_filter: [0.05, 0.1] diff --git a/scripts/run_benchmark/param_sweeps_full/comseg_params.yaml b/scripts/run_benchmark/param_sweeps_full/comseg_params.yaml new file mode 100644 index 000000000..3a5a07ea5 --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/comseg_params.yaml @@ -0,0 +1,32 @@ +# Parameter sweep for the comseg transcript assignment method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_transcript_assignment.qmd +# from the sweep results in /Users/daria.romanovskaia/projects/txsim_results/param_sweep_tests/transcript_assignment/comseg +# +# Selection: the 3 parameter families that reach the Pareto front (max cells, +# max negative marker purity) in the most datasets, tie-broken by how many datasets +# they beat `default` in and then by Euclidean distance from `default`. Within each +# family: every value on the front in >=1 dataset, plus the highest-distance value, +# topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total comseg variants = 1 default + 7 swept values. +parameters: + comseg: + default: + mean_cell_diameter: 15.0 + max_cell_radius: 25.0 + alpha: 0.5 + min_rna_per_cell: 5 + norm_vector: false + allow_disconnected_polygon: true + sweep: + # on front in 1/3 datasets, beats default in 1; picked: 10.0 (front + max distance), 20.0 (on front) + mean_cell_diameter: [10.0, 20.0] + # on front in 1/3 datasets, beats default in 0; picked: 1.0 (front + max distance), 0.75 (on front), 0.25 (on front) + alpha: [0.25, 0.75, 1.0] + # on front in 0/3 datasets, beats default in 0; picked: 20 (max distance), 10 (distance top-up) + min_rna_per_cell: [10, 20] diff --git a/scripts/run_benchmark/param_sweeps_full/fastreseg_params.yaml b/scripts/run_benchmark/param_sweeps_full/fastreseg_params.yaml new file mode 100644 index 000000000..933261dd0 --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/fastreseg_params.yaml @@ -0,0 +1,30 @@ +# Parameter sweep for the fastreseg transcript assignment method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_transcript_assignment.qmd +# from the sweep results in /Users/daria.romanovskaia/projects/txsim_results/param_sweep_tests/transcript_assignment/fastreseg +# +# Selection: the 3 parameter families that reach the Pareto front (max cells, +# max negative marker purity) in the most datasets, tie-broken by how many datasets +# they beat `default` in and then by Euclidean distance from `default`. Within each +# family: every value on the front in >=1 dataset, plus the highest-distance value, +# topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total fastreseg variants = 1 default + 0 swept values. +parameters: + fastreseg: + default: + molecular_distance_cutoff: 2.7 + flagCell_lrtest_cutoff: 5 + svmClass_score_cutoff: -2 + cutoff_spatialMerge: 0.5 + sweep: + # on front in 1/3 datasets, beats default in 0; picked: + cutoff_spatialMerge: [] + # on front in 1/3 datasets, beats default in 0; picked: + flagCell_lrtest_cutoff: [] + # on front in 1/3 datasets, beats default in 0; picked: + molecular_distance_cutoff: [] diff --git a/scripts/run_benchmark/param_sweeps_full/moscot_params.yaml b/scripts/run_benchmark/param_sweeps_full/moscot_params.yaml new file mode 100644 index 000000000..26e1b03ec --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/moscot_params.yaml @@ -0,0 +1,32 @@ +# Parameter sweep for the moscot cell type annotation method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_cell_type_annotation.qmd +# from the sweep results in /Users/daria.romanovskaia/projects/txsim_results/param_sweep_tests/cell_type_annotation/moscot +# +# Selection: the 3 parameter families that reach the Pareto front (max per-cell-type +# co-expression similarity, max negative marker purity) in the most datasets, tie-broken by +# how many datasets they beat `default` in and then by Euclidean distance from `default`. +# Within each family: every value on the front in >=1 dataset, plus the highest-distance +# value, topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total moscot variants = 1 default + 7 swept values. +parameters: + moscot: + default: + alpha: 0.8 + epsilon: 0.01 + tau: 0.3 + rank: 500 + batch_size: 1024 + mapping_mode: "max" + sweep: + # on front in 0/3 datasets, beats default in 2; picked: sum (max distance) + mapping_mode: ["sum"] + # on front in 0/3 datasets, beats default in 2; picked: 0.1 (max distance), 0.001 (distance top-up), 0.05 (distance top-up) + epsilon: [0.001, 0.05, 0.1] + # on front in 0/3 datasets, beats default in 1; picked: 0.5 (max distance), 0.7 (distance top-up), 0.9 (distance top-up) + alpha: [0.5, 0.7, 0.9] diff --git a/scripts/run_benchmark/param_sweeps_full/pciseq_params.yaml b/scripts/run_benchmark/param_sweeps_full/pciseq_params.yaml new file mode 100644 index 000000000..347928af0 --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/pciseq_params.yaml @@ -0,0 +1,31 @@ +# Parameter sweep for the pciseq transcript assignment method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_transcript_assignment.qmd +# from the sweep results in /Users/daria.romanovskaia/projects/txsim_results/param_sweep_tests/transcript_assignment/pciseq +# +# Selection: the 3 parameter families that reach the Pareto front (max cells, +# max negative marker purity) in the most datasets, tie-broken by how many datasets +# they beat `default` in and then by Euclidean distance from `default`. Within each +# family: every value on the front in >=1 dataset, plus the highest-distance value, +# topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total pciseq variants = 1 default + 8 swept values. +parameters: + pciseq: + default: + InsideCellBonus: 2 + MisreadDensity: 1e-05 + nNeighbors: 3 + Inefficiency: 0.2 + rGene: 20 + sweep: + # on front in 3/3 datasets, beats default in 2; picked: 0.001 (front + max distance), 1.0E-4 (on front), 1.0E-6 (distance top-up) + MisreadDensity: [1.0E-6, 1.0E-4, 0.001] + # on front in 2/3 datasets, beats default in 2; picked: 0 (front + max distance), 1 (on front), 6 (distance top-up) + InsideCellBonus: [0, 1, 6] + # on front in 1/3 datasets, beats default in 2; picked: 40 (front + max distance), 10 (distance top-up) + rGene: [10, 40] diff --git a/scripts/run_benchmark/param_sweeps_full/proseg_params.yaml b/scripts/run_benchmark/param_sweeps_full/proseg_params.yaml new file mode 100644 index 000000000..29a3cce10 --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/proseg_params.yaml @@ -0,0 +1,30 @@ +# Parameter sweep for the proseg transcript assignment method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_transcript_assignment.qmd +# from the sweep results in /Users/daria.romanovskaia/projects/txsim_results/param_sweep_tests/transcript_assignment/proseg +# +# Selection: the 3 parameter families that reach the Pareto front (max cells, +# max negative marker purity) in the most datasets, tie-broken by how many datasets +# they beat `default` in and then by Euclidean distance from `default`. Within each +# family: every value on the front in >=1 dataset, plus the highest-distance value, +# topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total proseg variants = 1 default + 7 swept values. +parameters: + proseg: + default: + cell_compactness: 0.04 + nuclear_reassignment_prob: 0.2 + diffusion_probability: 0.2 + ncomponents: 10 + sweep: + # on front in 1/3 datasets, beats default in 3; picked: 0.03 (front + max distance), 0.06 (distance top-up), 0.02 (distance top-up) + cell_compactness: [0.02, 0.03, 0.06] + # on front in 1/3 datasets, beats default in 2; picked: 0.5 (front + max distance), 0.05 (distance top-up) + nuclear_reassignment_prob: [0.05, 0.5] + # on front in 1/3 datasets, beats default in 1; picked: 15 (on front), 5 (max distance) + ncomponents: [5, 15] diff --git a/scripts/run_benchmark/param_sweeps_full/resolvi_correction_params.yaml b/scripts/run_benchmark/param_sweeps_full/resolvi_correction_params.yaml new file mode 100644 index 000000000..fb2e1cf42 --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/resolvi_correction_params.yaml @@ -0,0 +1,30 @@ +# Parameter sweep for the resolvi_correction expression correction method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_expression_correction.qmd +# from the sweep results in /Users/daria.romanovskaia/projects/txsim_results/param_sweep_tests/expression_correction/resolvi_correction +# +# Selection: the 3 parameter families that reach the Pareto front (max per-cell-type +# co-expression similarity, max negative marker purity) in the most datasets, tie-broken by +# how many datasets they beat `default` in and then by Euclidean distance from `default`. +# Within each family: every value on the front in >=1 dataset, plus the highest-distance +# value, topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total resolvi_correction variants = 1 default + 4 swept values. +parameters: + resolvi_correction: + default: + celltype_key: "cell_type" + n_hidden: 32 + encode_covariates: false + downsample_counts: true + sweep: + # on front in 0/3 datasets, beats default in 3; picked: 128 (max distance), 64 (distance top-up) + n_hidden: [64, 128] + # on front in 0/3 datasets, beats default in 2; picked: false (max distance) + downsample_counts: [false] + # on front in 0/3 datasets, beats default in 1; picked: true (max distance) + encode_covariates: [true] diff --git a/scripts/run_benchmark/param_sweeps_full/run_full_cellposev4_nebius.sh b/scripts/run_benchmark/param_sweeps_full/run_full_cellposev4_nebius.sh index 1f2755d51..dddc9269b 100644 --- a/scripts/run_benchmark/param_sweeps_full/run_full_cellposev4_nebius.sh +++ b/scripts/run_benchmark/param_sweeps_full/run_full_cellposev4_nebius.sh @@ -28,10 +28,10 @@ set -e resources_s3=/scratch/task_ist_preprocessing/datasets publish_dir="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_cellposev4_full_sweep" -# The sweep lives in a committed file, read from GitHub at runtime. $params_branch -# defaults to the branch you are on; the file must be pushed there on GitHub. +# The sweep lives in a committed file, read from GitHub at runtime from the `main` +# branch; the file must be committed AND pushed to main on GitHub before launching. params_repo="openproblems-bio/task_ist_preprocessing" -params_branch="$(git rev-parse --abbrev-ref HEAD)" +params_branch="main" params_url="https://raw.githubusercontent.com/${params_repo}/${params_branch}/scripts/run_benchmark/param_sweeps_full/cellposev4_params.yaml" cat > /tmp/params_settings_cellposev4_full.yaml << HERE diff --git a/scripts/run_benchmark/param_sweeps_full/run_full_resolvi_correction_nebius.sh b/scripts/run_benchmark/param_sweeps_full/run_full_resolvi_correction_nebius.sh new file mode 100644 index 000000000..4b4dc6dd4 --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/run_full_resolvi_correction_nebius.sh @@ -0,0 +1,100 @@ +#!/bin/bash + +# Nebius FULL run: all default methods + resolVI (resolvi_correction) expression correction, +# driven by the FOLLOW-UP (narrowed, Pareto-selected) parameter sweep, on the FULL datasets +# and the FULL (non-test) labels_nebius.config. +# +# Sweep file: param_sweeps_full/resolvi_correction_params.yaml (1 default + 4 swept = 5 +# variants; AUTO-GENERATED from the test-resource sweep results by +# results/param_sweeps/Yaml_to_heatmap_expression_correction.qmd — do not hand-edit here). +# Small/test sibling: ../param_sweep/run_test_resolvi_correction_nebius.sh +# +# resolVI is a scvi-tools VAE (deep model); its config carries the `gpu` label, routed by +# src/base/labels_nebius.config's gpu nodeSelector. The expression-correction stage lists BOTH +# the stage default (no_correction) AND resolvi_correction so the sweep is scored against the +# uncorrected baseline; every OTHER stage stays on its single default (resolvi_correction must +# be the only non-default thing in the pipeline). +# +# PARAMS-FILE CAVEAT: `method_parameters_yaml` is opened by the WORKFLOW at runtime on the +# cloud (readYaml -> Nextflow file()), which stages http(s):// but NOT a launch-host local +# path. So the sweep is read from GitHub via its raw URL => it must be committed AND PUSHED +# to $params_branch before launching. (Independent of --revision, which selects pipeline CODE.) + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# FULL datasets on the Nebius shared /scratch mount (not the small S3 test set). +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_resolvi_correction_full_sweep" + +# The sweep lives in a committed file, read from GitHub at runtime from the `main` +# branch; the file must be committed AND pushed to main on GitHub before launching. +params_repo="openproblems-bio/task_ist_preprocessing" +params_branch="main" +params_url="https://raw.githubusercontent.com/${params_repo}/${params_branch}/scripts/run_benchmark/param_sweeps_full/resolvi_correction_params.yaml" + +cat > /tmp/params_settings_resolvi_correction_full.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco +expression_correction_methods: + - no_correction + - resolvi_correction +gene_efficiency_correction_methods: + - no_correction +method_parameters_yaml: $params_url +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_resolvi_correction_full.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_resolvi_correction_full.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir" +HERE + +# Fail early with a clear message if the params file isn't reachable on GitHub yet. +if ! curl -fsSL -o /dev/null "$params_url"; then + echo "ERROR: params file not reachable at:" >&2 + echo " $params_url" >&2 + echo "Commit and push scripts/run_benchmark/param_sweeps_full/resolvi_correction_params.yaml to '$params_branch' first." >&2 + exit 1 +fi + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_resolvi_correction_full.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,full,resolvi_correction,gpu diff --git a/scripts/run_benchmark/param_sweeps_full/run_full_segger_nebius.sh b/scripts/run_benchmark/param_sweeps_full/run_full_segger_nebius.sh new file mode 100644 index 000000000..c310bdfff --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/run_full_segger_nebius.sh @@ -0,0 +1,100 @@ +#!/bin/bash + +# Nebius FULL run: all default methods + segger transcript assignment, driven by the +# FOLLOW-UP (narrowed, Pareto-selected) parameter sweep, on the FULL datasets and the +# FULL (non-test) labels_nebius.config. +# +# Sweep file: param_sweeps_full/segger_params.yaml (1 default + 6 swept = 7 variants; +# AUTO-GENERATED from the test-resource sweep results by +# results/param_sweeps/Yaml_to_heatmap_transcript_assignment.qmd — do not hand-edit here). +# Small/test sibling: ../param_sweep/run_test_segger_nebius.sh +# +# segger is GPU-only (RAPIDS GNN); its config carries the `gpuh100` label, routed by +# src/base/labels_nebius.config's gpuh100 nodeSelector (runAsUser:0 + /dev/shm). The stage +# default (basic_transcript_assignment) stays enabled so the run also produces the baseline +# the sweep is scored against; every OTHER stage stays on its single default (the workflow +# allows at most ONE non-default variant at a time, and segger is it here). +# +# PARAMS-FILE CAVEAT: `method_parameters_yaml` is opened by the WORKFLOW at runtime on the +# cloud (readYaml -> Nextflow file()), which stages http(s):// but NOT a launch-host local +# path. So the sweep is read from GitHub via its raw URL => it must be committed AND PUSHED +# to $params_branch before launching. (Independent of --revision, which selects pipeline CODE.) + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# FULL datasets on the Nebius shared /scratch mount (not the small S3 test set). +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_segger_full_sweep" + +# The sweep lives in a committed file, read from GitHub at runtime from the `main` +# branch; the file must be committed AND pushed to main on GitHub before launching. +params_repo="openproblems-bio/task_ist_preprocessing" +params_branch="main" +params_url="https://raw.githubusercontent.com/${params_repo}/${params_branch}/scripts/run_benchmark/param_sweeps_full/segger_params.yaml" + +cat > /tmp/params_settings_segger_full.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment + - segger +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco +expression_correction_methods: + - no_correction +gene_efficiency_correction_methods: + - no_correction +method_parameters_yaml: $params_url +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_segger_full.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_segger_full.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir" +HERE + +# Fail early with a clear message if the params file isn't reachable on GitHub yet. +if ! curl -fsSL -o /dev/null "$params_url"; then + echo "ERROR: params file not reachable at:" >&2 + echo " $params_url" >&2 + echo "Commit and push scripts/run_benchmark/param_sweeps_full/segger_params.yaml to '$params_branch' first." >&2 + exit 1 +fi + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_segger_full.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,full,segger diff --git a/scripts/run_benchmark/param_sweeps_full/run_full_stardist_nebius.sh b/scripts/run_benchmark/param_sweeps_full/run_full_stardist_nebius.sh index 4b6860bcf..c284ecad2 100644 --- a/scripts/run_benchmark/param_sweeps_full/run_full_stardist_nebius.sh +++ b/scripts/run_benchmark/param_sweeps_full/run_full_stardist_nebius.sh @@ -28,10 +28,10 @@ set -e resources_s3=/scratch/task_ist_preprocessing/datasets publish_dir="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_stardist_full_sweep" -# The sweep lives in a committed file, read from GitHub at runtime. $params_branch -# defaults to the branch you are on; the file must be pushed there on GitHub. +# The sweep lives in a committed file, read from GitHub at runtime from the `main` +# branch; the file must be committed AND pushed to main on GitHub before launching. params_repo="openproblems-bio/task_ist_preprocessing" -params_branch="$(git rev-parse --abbrev-ref HEAD)" +params_branch="main" params_url="https://raw.githubusercontent.com/${params_repo}/${params_branch}/scripts/run_benchmark/param_sweeps_full/stardist_params.yaml" cat > /tmp/params_settings_stardist_full.yaml << HERE diff --git a/scripts/run_benchmark/param_sweeps_full/run_full_tangram_nebius.sh b/scripts/run_benchmark/param_sweeps_full/run_full_tangram_nebius.sh new file mode 100644 index 000000000..0b6c099e3 --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/run_full_tangram_nebius.sh @@ -0,0 +1,100 @@ +#!/bin/bash + +# Nebius FULL run: all default methods + Tangram (tangram) cell-type annotation, driven by +# the FOLLOW-UP (narrowed, Pareto-selected) parameter sweep, on the FULL datasets and the +# FULL (non-test) labels_nebius.config. +# +# Sweep file: param_sweeps_full/tangram_params.yaml (1 default + 4 swept = 5 variants; +# AUTO-GENERATED from the test-resource sweep results by +# results/param_sweeps/Yaml_to_heatmap_cell_type_annotation.qmd — do not hand-edit here). +# Small/test sibling: ../param_sweep/run_test_tangram_nebius.sh +# +# Tangram is a torch deep-learning mapping method (GPU); its config carries the `gpuh100` +# label, routed by src/base/labels_nebius.config's gpuh100 nodeSelector. The annotation stage +# keeps its default method `tacco` alongside `tangram` so the run has a baseline to compare the +# tangram variants against; every other stage stays on its single default (the swept method +# must be the only non-default thing). +# +# PARAMS-FILE CAVEAT: `method_parameters_yaml` is opened by the WORKFLOW at runtime on the +# cloud (readYaml -> Nextflow file()), which stages http(s):// but NOT a launch-host local +# path. So the sweep is read from GitHub via its raw URL => it must be committed AND PUSHED +# to $params_branch before launching. (Independent of --revision, which selects pipeline CODE.) + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# FULL datasets on the Nebius shared /scratch mount (not the small S3 test set). +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_tangram_full_sweep" + +# The sweep lives in a committed file, read from GitHub at runtime from the `main` +# branch; the file must be committed AND pushed to main on GitHub before launching. +params_repo="openproblems-bio/task_ist_preprocessing" +params_branch="main" +params_url="https://raw.githubusercontent.com/${params_repo}/${params_branch}/scripts/run_benchmark/param_sweeps_full/tangram_params.yaml" + +cat > /tmp/params_settings_tangram_full.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco + - tangram +expression_correction_methods: + - no_correction +gene_efficiency_correction_methods: + - no_correction +method_parameters_yaml: $params_url +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_tangram_full.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_tangram_full.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir" +HERE + +# Fail early with a clear message if the params file isn't reachable on GitHub yet. +if ! curl -fsSL -o /dev/null "$params_url"; then + echo "ERROR: params file not reachable at:" >&2 + echo " $params_url" >&2 + echo "Commit and push scripts/run_benchmark/param_sweeps_full/tangram_params.yaml to '$params_branch' first." >&2 + exit 1 +fi + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_tangram_full.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,full,tangram,gpu diff --git a/scripts/run_benchmark/param_sweeps_full/run_full_watershed_nebius.sh b/scripts/run_benchmark/param_sweeps_full/run_full_watershed_nebius.sh index d5397acb8..af549342d 100644 --- a/scripts/run_benchmark/param_sweeps_full/run_full_watershed_nebius.sh +++ b/scripts/run_benchmark/param_sweeps_full/run_full_watershed_nebius.sh @@ -28,10 +28,10 @@ set -e resources_s3=/scratch/task_ist_preprocessing/datasets publish_dir="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_watershed_full_sweep" -# The sweep lives in a committed file, read from GitHub at runtime. $params_branch -# defaults to the branch you are on; the file must be pushed there on GitHub. +# The sweep lives in a committed file, read from GitHub at runtime from the `main` +# branch; the file must be committed AND pushed to main on GitHub before launching. params_repo="openproblems-bio/task_ist_preprocessing" -params_branch="$(git rev-parse --abbrev-ref HEAD)" +params_branch="main" params_url="https://raw.githubusercontent.com/${params_repo}/${params_branch}/scripts/run_benchmark/param_sweeps_full/watershed_params.yaml" cat > /tmp/params_settings_watershed_full.yaml << HERE diff --git a/scripts/run_benchmark/param_sweeps_full/segger_params.yaml b/scripts/run_benchmark/param_sweeps_full/segger_params.yaml new file mode 100644 index 000000000..09790f57e --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/segger_params.yaml @@ -0,0 +1,30 @@ +# Parameter sweep for the segger transcript assignment method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_transcript_assignment.qmd +# from the sweep results in /Users/daria.romanovskaia/projects/txsim_results/param_sweep_tests/transcript_assignment/segger +# +# Selection: the 3 parameter families that reach the Pareto front (max cells, +# max negative marker purity) in the most datasets, tie-broken by how many datasets +# they beat `default` in and then by Euclidean distance from `default`. Within each +# family: every value on the front in >=1 dataset, plus the highest-distance value, +# topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total segger variants = 1 default + 6 swept values. +parameters: + segger: + default: + n_epochs: 20 + prediction_graph_buffer_ratio: 0.05 + prediction_mode: "cell" + node_representation_dim: 128 + sweep: + # on front in 1/3 datasets, beats default in 2; picked: 0.1 (on front), 0.25 (max distance), 0.5 (distance top-up) + prediction_graph_buffer_ratio: [0.1, 0.25, 0.5] + # on front in 0/3 datasets, beats default in 2; picked: 60 (max distance), 40 (distance top-up) + n_epochs: [40, 60] + # on front in 0/3 datasets, beats default in 1; picked: nucleus (max distance) + prediction_mode: ["nucleus"] diff --git a/scripts/run_benchmark/param_sweeps_full/split_params.yaml b/scripts/run_benchmark/param_sweeps_full/split_params.yaml new file mode 100644 index 000000000..9359146de --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/split_params.yaml @@ -0,0 +1,32 @@ +# Parameter sweep for the split expression correction method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_expression_correction.qmd +# from the sweep results in /Users/daria.romanovskaia/projects/txsim_results/param_sweep_tests/expression_correction/split +# +# Selection: the 3 parameter families that reach the Pareto front (max per-cell-type +# co-expression similarity, max negative marker purity) in the most datasets, tie-broken by +# how many datasets they beat `default` in and then by Euclidean distance from `default`. +# Within each family: every value on the front in >=1 dataset, plus the highest-distance +# value, topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total split variants = 1 default + 6 swept values. +parameters: + split: + default: + gene_cutoff: 0.0 + fc_cutoff: 0.1 + gene_cutoff_reg: 0.0 + fc_cutoff_reg: 0.1 + umi_min: 20 + umi_min_sigma: 20 + sweep: + # on front in 2/3 datasets, beats default in 3; picked: 100 (front + max distance), 50 (distance top-up) + umi_min: [50, 100] + # on front in 1/3 datasets, beats default in 2; picked: 1.0E-4 (on front), 2.0E-4 (max distance) + gene_cutoff_reg: [1.0E-4, 2.0E-4] + # on front in 0/3 datasets, beats default in 3; picked: 0.75 (max distance), 0.4 (distance top-up) + fc_cutoff_reg: [0.4, 0.75] diff --git a/scripts/run_benchmark/param_sweeps_full/tangram_params.yaml b/scripts/run_benchmark/param_sweeps_full/tangram_params.yaml new file mode 100644 index 000000000..634153dbb --- /dev/null +++ b/scripts/run_benchmark/param_sweeps_full/tangram_params.yaml @@ -0,0 +1,26 @@ +# Parameter sweep for the tangram cell type annotation method - FOLLOW-UP (narrowed) sweep. +# +# AUTO-GENERATED - do not hand-edit; regenerate by rendering +# results/param_sweeps/Yaml_to_heatmap_cell_type_annotation.qmd +# from the sweep results in /Users/daria.romanovskaia/projects/txsim_results/param_sweep_tests/cell_type_annotation/tangram +# +# Selection: the 3 parameter families that reach the Pareto front (max per-cell-type +# co-expression similarity, max negative marker purity) in the most datasets, tie-broken by +# how many datasets they beat `default` in and then by Euclidean distance from `default`. +# Within each family: every value on the front in >=1 dataset, plus the highest-distance +# value, topped up by distance to 3 values where the family has that many. +# +# Consumed by the run_benchmark workflow via the `method_parameters_yaml` setting +# (src/workflows/run_benchmark/main.nf): one "default" variant from `default:`, +# plus one variant per value in each `sweep:` list, varying a SINGLE arg at a time. +# Total tangram variants = 1 default + 4 swept values. +parameters: + tangram: + default: + mode: "clusters" + num_epochs: 1000 + sweep: + # on front in 1/3 datasets, beats default in 1; picked: cells (front + max distance) + mode: ["cells"] + # on front in 0/3 datasets, beats default in 1; picked: 3000 (max distance), 2000 (distance top-up), 100 (distance top-up) + num_epochs: [100, 2000, 3000] diff --git a/scripts/run_benchmark/run_allen_merfish_nebius.sh b/scripts/run_benchmark/run_allen_merfish_nebius.sh new file mode 100644 index 000000000..e1e388ee5 --- /dev/null +++ b/scripts/run_benchmark/run_allen_merfish_nebius.sh @@ -0,0 +1,105 @@ +#!/bin/bash + +# Run the FULL benchmark (all methods) on the Allen Brain Cell Atlas (ABCA) MERFISH +# datasets ONLY. +# +# Same method settings as run_full_nebius.sh; the only difference is that +# `input_states` is globbed to the ABCA MERFISH combined datasets instead of all +# datasets under the resources dir. These are published by +# scripts/create_resources/combine/process_datasets_allen_nebius.sh (ids +# "allen_brain_cell_atlas_merfish_combined/mouse{1,2,3,4}_{coronal,sagittal}/rep1"), +# so their state.yaml files live under +# $resources_s3/allen_brain_cell_atlas_merfish_combined/**/state.yaml. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_allen_merfish" + +# restrict the benchmark to the ABCA MERFISH combined datasets only. +# The ** is expanded by Nextflow's glob (not the shell — it stays literal inside +# the heredoc below), matching all four mouse sections (mouse1_coronal, +# mouse2_coronal, mouse3_sagittal, mouse4_sagittal) under the combined folder. +dataset_glob="$resources_s3/allen_brain_cell_atlas_merfish_combined/**/state.yaml" + +cat > /tmp/params_settings_allen.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation + - cellpose + - cellposev4 + - binning + - stardist + - watershed +transcript_assignment_methods: + - basic_transcript_assignment + - baysor + - clustermap + - pciseq + - comseg + - proseg + - segger + - fastreseg +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume + - normalize_by_counts + - spanorm +celltype_annotation_methods: + - ssam + - tacco + - moscot + - mapmycells + - tangram + - singler + - rctd +expression_correction_methods: + - no_correction + - resolvi_correction + - split +gene_efficiency_correction_methods: + - no_correction +# - gene_efficiency_correction + +#method_parameters_yaml: /tmp/method_params.yaml +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_allen_benchmark.yaml << HERE +input_states: $dataset_glob +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_allen.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_allen_benchmark.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,allen_merfish diff --git a/scripts/run_benchmark/run_full_nebius.sh b/scripts/run_benchmark/run_full_nebius.sh new file mode 100644 index 000000000..07b402322 --- /dev/null +++ b/scripts/run_benchmark/run_full_nebius.sh @@ -0,0 +1,100 @@ +#!/bin/bash + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +#resources_s3=s3://openproblems-data/resources_test/task_ist_preprocessing +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)" + +cat > /tmp/params_settings.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation + - cellpose + - cellposev4 + - binning + - stardist + - watershed +transcript_assignment_methods: + - basic_transcript_assignment + - baysor + - clustermap + - pciseq + - comseg + - proseg + - segger + - fastreseg +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume + - normalize_by_counts + - spanorm +celltype_annotation_methods: + - ssam + - tacco + - moscot + - mapmycells + - tangram + - singler + - rctd +expression_correction_methods: + - no_correction + - resolvi_correction + - split +gene_efficiency_correction_methods: + - no_correction +# - gene_efficiency_correction + +#method_parameters_yaml: /tmp/method_params.yaml +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +# NOTE: this file needs to be made available on the seqera cloud workspace and the +# path needs to be added above (method_parameters_yaml) +#cat > /tmp/method_params.yaml << HERE +#parameters: +# binning: +# default: +# bin_size: 30 +# sweep: +# bin_size: [20, 30, 40] +#HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,test \ No newline at end of file diff --git a/scripts/run_benchmark/run_full_resolvi_nebius.sh b/scripts/run_benchmark/run_full_resolvi_nebius.sh new file mode 100644 index 000000000..778fb978b --- /dev/null +++ b/scripts/run_benchmark/run_full_resolvi_nebius.sh @@ -0,0 +1,109 @@ +#!/bin/bash + +# Run the FULL benchmark matrix across ALL datasets on Nebius, but with resolVI as the +# DEFAULT expression-correction method (run_full_nebius.sh uses `no_correction`). +# +# Modeled on run_full_nebius.sh: every per-stage sweep list is IDENTICAL, so the same +# star-matrix of segmentation / transcript-assignment / normalization / cell-type-annotation +# variants is explored. The ONLY change is the expression-correction step: +# run_full_nebius.sh : expr-corr default = no_correction (correction OFF by default) +# this script : expr-corr default = resolvi_correction (resolVI applied to every pipeline) +# +# HOW THE DEFAULT IS SET (same idiom as run_rctd_split_nebius.sh): +# `resolvi_correction` is placed in `default_methods` AND is the sole entry of +# `expression_correction_methods`, so it is the always-run backbone correction that every +# swept pipeline chains through. The expr-corr `no_correction` component is simply not +# selected at that stage, so it never runs there. `no_correction` stays in `default_methods` +# only as the gene-efficiency default -- a DIFFERENT component that happens to share the +# config name `no_correction`; it runs only at the gene-efficiency stage, the sole stage +# that still selects it. +# +# COST NOTE: resolVI is GPU + hightime + highmem (see its config `label: [hightime, highcpu, +# highmem, gpu]`; GPU tasks are routed by src/base/labels_nebius.config, same as run_gpu_nebius.sh). +# As the DEFAULT it now runs on EVERY pipeline in the matrix, so this schedules a LARGE number +# of GPU jobs -- considerably more expensive than run_full or run_gpu (which only sweep resolVI +# against the all-default backbone). Trim the upstream sweep lists if that is too heavy. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_resolvi" + +cat > /tmp/params_settings_resolvi.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - resolvi_correction + - no_correction +segmentation_methods: + - custom_segmentation + - cellpose + - cellposev4 + - binning + - stardist + - watershed +transcript_assignment_methods: + - basic_transcript_assignment + - baysor + - clustermap + - pciseq + - comseg + - proseg + - segger + - fastreseg +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume + - normalize_by_counts + - spanorm +celltype_annotation_methods: + - ssam + - tacco + - moscot + - mapmycells + - tangram + - singler + - rctd +expression_correction_methods: + - resolvi_correction +gene_efficiency_correction_methods: + - no_correction +# - gene_efficiency_correction +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_resolvi.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_resolvi.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_resolvi.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,resolvi diff --git a/scripts/run_benchmark/run_full_split_nebius.sh b/scripts/run_benchmark/run_full_split_nebius.sh new file mode 100644 index 000000000..7b5d88ab1 --- /dev/null +++ b/scripts/run_benchmark/run_full_split_nebius.sh @@ -0,0 +1,106 @@ +#!/bin/bash + +# Run the FULL benchmark matrix across ALL datasets on Nebius, but with SPLIT as the +# DEFAULT expression-correction method (run_full_nebius.sh uses `no_correction`). +# +# Modeled on run_full_nebius.sh: every per-stage sweep list is IDENTICAL, so the same +# star-matrix of segmentation / transcript-assignment / normalization / cell-type-annotation +# variants is explored. The ONLY change is the expression-correction step: +# run_full_nebius.sh : expr-corr default = no_correction (correction OFF by default) +# this script : expr-corr default = split (SPLIT applied to every pipeline) +# +# HOW THE DEFAULT IS SET (same idiom as run_rctd_split_nebius.sh): +# `split` is placed in `default_methods` AND is the sole entry of +# `expression_correction_methods`, so it is the always-run backbone correction that every +# swept pipeline chains through. The expr-corr `no_correction` component is simply not +# selected at that stage, so it never runs there. `no_correction` stays in `default_methods` +# only as the gene-efficiency default -- a DIFFERENT component that happens to share the +# config name `no_correction`; it runs only at the gene-efficiency stage, the sole stage +# that still selects it. +# +# COST NOTE: SPLIT (RCTD-based, CPU) now runs on EVERY pipeline in the matrix, not just the +# all-default backbone, so this run does substantially more correction work than run_full. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_split" + +cat > /tmp/params_settings_split.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - split + - no_correction +segmentation_methods: + - custom_segmentation + - cellpose + - cellposev4 + - binning + - stardist + - watershed +transcript_assignment_methods: + - basic_transcript_assignment + - baysor + - clustermap + - pciseq + - comseg + - proseg + - segger + - fastreseg +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume + - normalize_by_counts + - spanorm +celltype_annotation_methods: + - ssam + - tacco + - moscot + - mapmycells + - tangram + - singler + - rctd +expression_correction_methods: + - split +gene_efficiency_correction_methods: + - no_correction +# - gene_efficiency_correction +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_split.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_split.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_split.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,split diff --git a/scripts/run_benchmark/run_gpu_nebius.sh b/scripts/run_benchmark/run_gpu_nebius.sh new file mode 100644 index 000000000..283e8b0b2 --- /dev/null +++ b/scripts/run_benchmark/run_gpu_nebius.sh @@ -0,0 +1,69 @@ +#!/bin/bash + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +resources_dir="/scratch/task_ist_preprocessing/datasets" +publish_dir="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_gpu" + +cat > /tmp/params_settings_gpu.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation + - cellpose + - cellposev4 + - stardist +transcript_assignment_methods: + - basic_transcript_assignment +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco + - tangram + - moscot +expression_correction_methods: + - no_correction + - resolvi_correction +gene_efficiency_correction_methods: + - no_correction + # - gene_efficiency_correction +HERE + +cat > /tmp/params_gpu.yaml << HERE +input_states: $resources_dir/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_gpu.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_gpu.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,gpu \ No newline at end of file diff --git a/scripts/run_benchmark/run_mpii_nebius.sh b/scripts/run_benchmark/run_mpii_nebius.sh new file mode 100644 index 000000000..eba3f2d4d --- /dev/null +++ b/scripts/run_benchmark/run_mpii_nebius.sh @@ -0,0 +1,103 @@ +#!/bin/bash + +# Run the FULL benchmark (all methods) on the MPII + LTX human-lung Xenium datasets. +# +# Same method settings as run_full_nebius.sh; the only difference is that +# `input_states` is globbed to the MPII and LTX combined datasets instead of all +# datasets under the resources dir. These are published by +# scripts/create_resources/combine/process_datasets_mpii_nebius.sh (id +# "mpii_human_lung_xenium_combined/978_reg1") and process_datasets_ltx_nebius.sh +# (ids "ltx_human_lung_xenium_combined/{B_AK-16-12103,L_AK14_14254}"), so their +# state.yaml files live under +# $resources_s3/{mpii,ltx}_human_lung_xenium_combined/**/state.yaml. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_mpii_ltx" + +# restrict the benchmark to the MPII and LTX combined datasets only. +# The brace pattern is expanded by Nextflow's glob (not the shell — it stays +# literal inside the heredoc below), matching both the MPII sample and the two +# LTX samples under their respective combined-dataset folders. +dataset_glob="$resources_s3/{mpii_human_lung_xenium_combined,ltx_human_lung_xenium_combined}/**/state.yaml" + +cat > /tmp/params_settings_mpii.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation + - cellpose + - cellposev4 + - binning + - stardist + - watershed +transcript_assignment_methods: + - basic_transcript_assignment + - baysor + - clustermap + - pciseq + - comseg + - proseg +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume + - normalize_by_counts + - spanorm +celltype_annotation_methods: + - ssam + - tacco + - moscot + - mapmycells + - tangram + - singler + - rctd +expression_correction_methods: + - no_correction + - resolvi_correction + - split +gene_efficiency_correction_methods: + - no_correction + - gene_efficiency_correction + +#method_parameters_yaml: /tmp/method_params.yaml +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_mpii_benchmark.yaml << HERE +input_states: $dataset_glob +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_mpii.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_mpii_benchmark.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,mpii,ltx diff --git a/scripts/run_benchmark/run_pciseq_full_nebius.sh b/scripts/run_benchmark/run_pciseq_full_nebius.sh new file mode 100644 index 000000000..cce4db559 --- /dev/null +++ b/scripts/run_benchmark/run_pciseq_full_nebius.sh @@ -0,0 +1,83 @@ +#!/bin/bash + +# FULL pciSeq run across ALL datasets on Nebius — pciSeq isolated as the single +# non-default method. +# +# PURPOSE: run pciSeq transcript assignment on every processed dataset on its own, rather +# than as one variant inside the big star-matrix of run_full_nebius.sh. Isolating it makes +# it easy to (re)launch / watch / triage just pciSeq (e.g. after the empty-segmentation guard +# + re-processing fix) without re-running the whole matrix. +# +# Scope: pciSeq is the ONLY non-default method (the workflow allows at most one non-default +# per pipeline), so per dataset this produces the all-default backbone + the pciSeq variant. +# Runs on ALL datasets under $resources_s3 (the broad **/state.yaml glob, no `filter`). +# +# PREREQUISITE: the datasets under $resources_s3 must already be processed (this run consumes +# existing state.yaml files; it does NOT re-run process_datasets). +# +# COST NOTE: pciSeq is CPU (its config `label: [veryhightime, midcpu, highmem]`) — no GPU +# nodes. One pciSeq task per dataset, each up to the veryhightime (24h) ceiling on large panels. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Full datasets live on the Nebius shared /scratch mount (not the small S3 test set). +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_pciseq_full" + +cat > /tmp/params_settings_pciseq_full.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment + - pciseq +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco +expression_correction_methods: + - no_correction +gene_efficiency_correction_methods: + - no_correction +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this). +cat > /tmp/params_pciseq_full.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_pciseq_full.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_pciseq_full.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,full,pciseq diff --git a/scripts/run_benchmark/run_rctd_split_nebius.sh b/scripts/run_benchmark/run_rctd_split_nebius.sh new file mode 100644 index 000000000..ee19f3b77 --- /dev/null +++ b/scripts/run_benchmark/run_rctd_split_nebius.sh @@ -0,0 +1,77 @@ +#!/bin/bash + +# Run a SINGLE-pipeline benchmark across ALL datasets on Nebius: +# - cell type annotation : RCTD +# - expression correction: SPLIT +# - every other step uses its DEFAULT method (no method sweeps) +# +# Modeled on run_full_nebius.sh, but each per-stage method list is a singleton +# so the star-matrix collapses to exactly one pipeline per dataset. For that +# single chain to survive every stage, ALL of its methods must appear in +# `default_methods` (the star-matrix truncates once any non-default method is +# used upstream). `no_correction` is the gene-efficiency default; it only runs +# at the gene-eff stage because that is the only stage that selects it, so it is +# harmless alongside `split` in default_methods. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_rctd_split" + +cat > /tmp/params_settings_rctd_split.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - rctd + - split + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - rctd +expression_correction_methods: + - split +gene_efficiency_correction_methods: + - no_correction +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_rctd_split.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_rctd_split.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_rctd_split.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,rctd,split diff --git a/scripts/run_benchmark/run_segger_fastreseg_comseg_moscot_nebius.sh b/scripts/run_benchmark/run_segger_fastreseg_comseg_moscot_nebius.sh new file mode 100644 index 000000000..c1163861e --- /dev/null +++ b/scripts/run_benchmark/run_segger_fastreseg_comseg_moscot_nebius.sh @@ -0,0 +1,93 @@ +#!/bin/bash + +# Full (ALL datasets) run for the segger, fastreseg, comseg (transcript-assignment) +# and moscot (cell-type-annotation) methods. Runs over the full datasets on the +# Nebius shared /scratch mount, using the DEFAULT method at every stage, and adds: +# * segger, fastreseg, comseg to the transcript-assignment stage +# * moscot to the cell-type-annotation stage +# +# The benchmark workflow enforces "at most one non-default method per pipeline" +# (see expandChannelWithParameterSets in src/workflows/run_benchmark/main.nf), +# i.e. it varies exactly one stage at a time from the defaults. So, per dataset, +# this launch produces these runs: +# - all-default backbone +# - + segger (transcript assignment; rest default, incl. tacco) +# - + fastreseg (transcript assignment; rest default, incl. tacco) +# - + comseg (transcript assignment; rest default, incl. tacco) -> +# - + moscot (cell-type annotation; rest default, incl. basic_transcript_assignment) +# There is no segger+moscot etc. combination — that would be two non-default steps. +# +# Resource routing (all handled by src/base/labels_nebius.config, one launch): +# * segger -> GPU (L40S node group; [hightime, midcpu, highmem, gpu]) +# * moscot -> GPU (H100 node group; [hightime, midcpu, veryhighmem, gpuh100]) +# * fastreseg -> CPU ([midtime, midcpu, midmem]) +# * comseg -> CPU ([hightime, midcpu, highmem]) + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Full datasets live on the Nebius shared /scratch mount (not the small S3 test set). +#resources_s3=s3://openproblems-data/resources_test/task_ist_preprocessing +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_segger_fastreseg_comseg_moscot" + +cat > /tmp/params_settings_segger_fastreseg_comseg_moscot.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment + - segger + - fastreseg +# - comseg +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco +# - moscot +expression_correction_methods: + - no_correction +gene_efficiency_correction_methods: + - no_correction + # - gene_efficiency_correction +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_segger_fastreseg_comseg_moscot.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_segger_fastreseg_comseg_moscot.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_segger_fastreseg_comseg_moscot.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,full,segger,fastreseg,comseg,moscot diff --git a/scripts/run_benchmark/run_segger_fastreseg_resolvi_nebius.sh b/scripts/run_benchmark/run_segger_fastreseg_resolvi_nebius.sh new file mode 100644 index 000000000..475a10595 --- /dev/null +++ b/scripts/run_benchmark/run_segger_fastreseg_resolvi_nebius.sh @@ -0,0 +1,94 @@ +#!/bin/bash + +# segger + fastreseg transcript-assignment run across ALL datasets on Nebius, with resolVI as the +# DEFAULT expression-correction method (the sibling run_segger_fastreseg_comseg_moscot_nebius.sh +# uses `no_correction`). +# +# Same shape as that script: the benchmark enforces "at most one non-default method per +# pipeline" (expandChannelWithParameterSets in src/workflows/run_benchmark/main.nf), so per +# dataset this launch produces: +# - all-default backbone (basic_transcript_assignment; rest default) + resolVI +# - + segger (transcript assignment; rest default) + resolVI +# - + fastreseg (transcript assignment; rest default) + resolVI +# resolVI is applied to EVERY one of these because it is the always-run backbone correction. +# +# HOW THE DEFAULT IS SET (same idiom as run_full_resolvi_nebius.sh / run_rctd_split_nebius.sh): +# `resolvi_correction` is in `default_methods` AND is the sole entry of +# `expression_correction_methods`, so it is the always-run backbone correction. The expr-corr +# `no_correction` component is not selected at that stage, so it never runs there; +# `no_correction` stays in `default_methods` only as the gene-efficiency default (a different +# component that shares the config name). +# +# Resource routing (all via src/base/labels_nebius.config, one launch): +# * segger -> GPU (L40S node group; [hightime, midcpu, highmem, gpu]) +# * fastreseg -> CPU ([midtime, midcpu, midmem]) +# * resolvi -> GPU ([hightime, highcpu, highmem, gpu]) -- runs on EVERY branch (default), +# so the segger branch is GPU at BOTH transcript-assignment and correction. + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Full datasets live on the Nebius shared /scratch mount (not the small S3 test set). +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_segger_fastreseg_resolvi" + +cat > /tmp/params_settings_segger_fastreseg_resolvi.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - resolvi_correction + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment + - segger + - fastreseg +# - comseg +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco +expression_correction_methods: + - resolvi_correction +gene_efficiency_correction_methods: + - no_correction + # - gene_efficiency_correction +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_segger_fastreseg_resolvi.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_segger_fastreseg_resolvi.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_segger_fastreseg_resolvi.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,segger,fastreseg,resolvi diff --git a/scripts/run_benchmark/run_segger_fastreseg_split_nebius.sh b/scripts/run_benchmark/run_segger_fastreseg_split_nebius.sh new file mode 100644 index 000000000..952c5270f --- /dev/null +++ b/scripts/run_benchmark/run_segger_fastreseg_split_nebius.sh @@ -0,0 +1,92 @@ +#!/bin/bash + +# segger + fastreseg transcript-assignment run across ALL datasets on Nebius, with SPLIT as the +# DEFAULT expression-correction method (the sibling run_segger_fastreseg_comseg_moscot_nebius.sh +# uses `no_correction`). +# +# Same shape as that script: the benchmark enforces "at most one non-default method per +# pipeline" (expandChannelWithParameterSets in src/workflows/run_benchmark/main.nf), so per +# dataset this launch produces: +# - all-default backbone (basic_transcript_assignment; rest default) + SPLIT +# - + segger (transcript assignment; rest default) + SPLIT +# - + fastreseg (transcript assignment; rest default) + SPLIT +# SPLIT is applied to EVERY one of these because it is the always-run backbone correction. +# +# HOW THE DEFAULT IS SET (same idiom as run_full_split_nebius.sh / run_rctd_split_nebius.sh): +# `split` is in `default_methods` AND is the sole entry of `expression_correction_methods`, +# so it is the always-run backbone correction. The expr-corr `no_correction` component is not +# selected at that stage, so it never runs there; `no_correction` stays in `default_methods` +# only as the gene-efficiency default (a different component that shares the config name). +# +# Resource routing (all via src/base/labels_nebius.config, one launch): +# * segger -> GPU (L40S node group; [hightime, midcpu, highmem, gpu]) +# * fastreseg -> CPU ([midtime, midcpu, midmem]) +# * split -> CPU (RCTD-based) + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Full datasets live on the Nebius shared /scratch mount (not the small S3 test set). +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_segger_fastreseg_split" + +cat > /tmp/params_settings_segger_fastreseg_split.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - split + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment + - segger + - fastreseg +# - comseg +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco +expression_correction_methods: + - split +gene_efficiency_correction_methods: + - no_correction + # - gene_efficiency_correction +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params_segger_fastreseg_split.yaml << HERE +input_states: $resources_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_segger_fastreseg_split.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_segger_fastreseg_split.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,segger,fastreseg,split diff --git a/scripts/run_benchmark/run_segger_full_validation_nebius.sh b/scripts/run_benchmark/run_segger_full_validation_nebius.sh new file mode 100644 index 000000000..18c7fca14 --- /dev/null +++ b/scripts/run_benchmark/run_segger_full_validation_nebius.sh @@ -0,0 +1,119 @@ +#!/bin/bash + +# FULL segger run on a few real datasets — validation for the transcript-overlap fix. +# +# PURPOSE: confirm whether the process_dataset crop fix (commit a459afca5, the +# transform-agnostic crop_points_by_global_xy that now crops transcripts to the SAME global +# box as the labels) actually eliminates the out-of-bounds (OOB) transcripts that drove +# segger's `in_bounds` workaround. If it does, segger's OOB-exclusion + row_index remap can be +# simplified to a plain edge clamp (like baysor/proseg). This run is the evidence for that +# decision — DO NOT remove the workaround until this run confirms OOB ~ 0 on real data. +# +# HOW TO READ THE RESULT: segger's script.py prints, per dataset, a line like +# "/ transcripts fall outside the HxW label image; excluding them ..." +# Pull it from each segger task's .command.log (Seqera UI or the work dir). Interpretation: +# * n_oob ~ 0 (only a handful, edge rounding) -> crop fix works; safe to simplify the guard. +# * n_oob still large (e.g. tens of %) -> either these datasets were NOT re-processed +# with a459afca5, or the fix is incomplete; +# KEEP the workaround. +# Also: a completed segger task (no empty-`bd` mid-training crash) is itself part of the signal. +# +# PREREQUISITE (important): the datasets under $resources_s3 must have been RE-PROCESSED with +# the fixed process_dataset (a459afca5). This run consumes already-processed state.yaml files; +# it does NOT re-run process_datasets. If /scratch still holds pre-fix datasets, the OOB count +# reflects the OLD crop, not the fix. +# +# Scope: segger is the ONLY non-default method (the workflow allows at most one non-default +# per pipeline), so per dataset this produces the all-default backbone + the segger variant. +# segger is GPU-only; the `gpuh100` label in its config + src/base/labels_nebius.config +# (runAsUser:0 + /dev/shm volume) pin it to the GPU node group — no GPU-specific change here +# (same mechanism as run_test_segger_nebius.sh / run_segger_fastreseg_comseg_moscot_nebius.sh). + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +# Full datasets live on the Nebius shared /scratch mount (not the small S3 test set). +resources_s3=/scratch/task_ist_preprocessing/datasets +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_segger_full_validation" + +# --- Datasets to validate on (edit this list to pick 2-3) ----------------------------------- +# Chosen to exercise the crop across BOTH transform types the overlap bug hit differently: +# * 2023_10x_mouse_brain_xenium_combined/rep3 -> Xenium / Scale; the exact dataset segger's +# NOTES measured 7.6M/19.7M (~38%) OOB on. +# * 2023_10x_human_breast_cancer_xenium_combined -> a second, larger Xenium (different panel). +# * 2022_vizgen_human_breast_cancer_merfish_combined/rep1 -> MERFISH / Affine (the OTHER crop +# regression path; the lung merfish is avoided +# because it has all-zero labels -> segger's +# empty-segmentation guard would fail fast). +# IDs are the dataset dir names under $resources_s3 (state.yaml lives at /state.yaml). +datasets=( + "2023_10x_mouse_brain_xenium_combined/rep3" + "2023_10x_human_breast_cancer_xenium_combined" + "2022_vizgen_human_breast_cancer_merfish_combined/rep1" +) + +# Build a findStates `--filter` regex (matched against the FULL state-file path with +# .matches(), so anchor with .* and end at .../state.yaml). IDs contain only [A-Za-z0-9_/], +# no regex metacharacters, so they need no escaping; only the literal dot in state.yaml does. +filter_ids=$(IFS='|'; echo "${datasets[*]}") +filter_regex=".*/datasets/(${filter_ids})/state\\.yaml" +echo "Filtering datasets with regex: $filter_regex" + +cat > /tmp/params_settings_segger_full.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation +transcript_assignment_methods: + - basic_transcript_assignment + - segger +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +celltype_annotation_methods: + - tacco +expression_correction_methods: + - no_correction +gene_efficiency_correction_methods: + - no_correction +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this). +# `filter` restricts the broad **/state.yaml glob to just the datasets listed above. +cat > /tmp/params_segger_full.yaml << HERE +input_states: $resources_s3/**/state.yaml +filter: '$filter_regex' +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings_segger_full.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params_segger_full.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,full,segger,validation diff --git a/scripts/run_benchmark/run_test_nebius.sh b/scripts/run_benchmark/run_test_nebius.sh new file mode 100644 index 000000000..ee9ccbb1e --- /dev/null +++ b/scripts/run_benchmark/run_test_nebius.sh @@ -0,0 +1,114 @@ +#!/bin/bash + +# get the root of the directory +REPO_ROOT=$(git rev-parse --show-toplevel) + +# ensure that the command below is run from the root of the repository +cd "$REPO_ROOT" + +set -e + +resources_test_s3="/scratch/task_ist_preprocessing/resources_test/task_ist_preprocessing/" +publish_dir_s3="/scratch/results/runs/$(date +%Y-%m-%d_%H-%M-%S)_test" + +cat > /tmp/params_settings.yaml << HERE +default_methods: + - custom_segmentation + - basic_transcript_assignment + - basic_count_aggregation + - basic_qc_filter + - alpha_shapes + - normalize_by_volume + - tacco + - no_correction +segmentation_methods: + - custom_segmentation +# - cellpose +# - cellposev4 +# - binning +# - stardist +# - watershed +transcript_assignment_methods: + - basic_transcript_assignment +# - baysor +# - clustermap +# - pciseq +# - comseg +# - proseg +count_aggregation_methods: + - basic_count_aggregation +qc_filtering_methods: + - basic_qc_filter +volume_calculation_methods: + - alpha_shapes +normalization_methods: + - normalize_by_volume +# - normalize_by_counts +# - spanorm +celltype_annotation_methods: +# - ssam + - tacco +# - moscot +# - mapmycells +# - tangram +# - singler +# - rctd +expression_correction_methods: + - no_correction +# - resolvi_correction +# - split +gene_efficiency_correction_methods: + - no_correction +# - gene_efficiency_correction +#method_parameters_yaml: /tmp/method_params.yaml +HERE + +# Write the parameters to file (input_states version, NOTE: enable `-entry_name auto` for this) +cat > /tmp/params.yaml << HERE +input_states: $resources_test_s3/**/state.yaml +rename_keys: 'input_sc:output_sc;input_sp:output_sp' +save_spatial_data: false +settings: '$(yq -o json /tmp/params_settings.yaml | jq -c .)' +output_state: "state.yaml" +publish_dir: "$publish_dir_s3" +HERE + +# # write the parameters to file (specific id version, NOTE: disable `-entry_name auto` for this) +# cat > /tmp/params.yaml << HERE +# id: mouse_brain_combined +# input_sc: $resources_test_s3/mouse_brain_combined/scrnaseq_reference.h5ad +# input_sp: $resources_test_s3/mouse_brain_combined/raw_ist.zarr +# save_spatial_data: false +# settings: '$(yq -o json /tmp/params_settings.yaml | jq -c .)' +# output_state: "state.yaml" +# publish_dir: $publish_dir_s3 +# HERE + +# NOTE: this file needs to be made available on the seqera cloud workspace and the +# path needs to be added above (method_parameters_yaml) +#cat > /tmp/method_params.yaml << HERE +#parameters: +# binning: +# default: +# bin_size: 30 +# sweep: +# bin_size: [20, 30, 40] +#HERE + + +tw launch https://github.com/openproblems-bio/task_ist_preprocessing.git \ + --revision build/main \ + --pull-latest \ + --main-script target/nextflow/workflows/run_benchmark/main.nf \ + --workspace 167877437119966 \ + --compute-env 5hfmdCBxMRd4nHZaJKYEQZ \ + --params-file /tmp/params.yaml \ + --entry-name auto \ + --config src/base/labels_nebius.config \ + --labels task_ist_preprocessing,test + +# aws s3 sync \ +# s3://openproblems-nextflow/temp/results \ +# temp_results \ +# --profile op \ +# --dryrun diff --git a/scripts/sync_scratch_datasets_to_s3.sh b/scripts/sync_scratch_datasets_to_s3.sh new file mode 100644 index 000000000..f85726dd6 --- /dev/null +++ b/scripts/sync_scratch_datasets_to_s3.sh @@ -0,0 +1,223 @@ +#!/usr/bin/env bash +# sync_scratch_datasets_to_s3.sh — push the combined datasets on the cluster +# scratch PVC up to a NEW `datasets/zarr3/` prefix in S3. +# +# SRC (cluster): /scratch/task_ist_preprocessing/datasets//... +# DST (S3): s3://openproblems-data/resources/task_ist_preprocessing/datasets/zarr3//... +# +# WHY A POD: the 375 GB of datasets live on the `tower-scratch` PVC (RWX, ns +# tower-nf, mounted at /scratch), which the launch host cannot see; and the AWS +# creds live in your local `op` profile, which the cluster does not have. So we +# bridge them: read the `op` creds locally, hand them to a short-lived pod (via a +# k8s Secret, so they are NOT baked into the pod spec or exec argv), and run +# `aws s3 sync` from inside that pod — data goes cluster -> S3 directly (never +# through your laptop). +# +# ADDITIVE + RESUMABLE: writes to a fresh `zarr3/` prefix and never uses +# --delete, so nothing already in S3 is touched. `aws s3 sync` skips files +# already uploaded with a matching size, so if the transfer drops just re-run +# `start` (or use `status` to watch the one already running). +# +# The sync runs DETACHED inside the pod (nohup -> a log on the PVC), so a dropped +# kubectl connection does not kill a multi-hour transfer. +# +# Usage: +# scripts/sync_scratch_datasets_to_s3.sh start [opts] # spawn pod + launch sync (default) +# scripts/sync_scratch_datasets_to_s3.sh status [opts] # pod state + live sync log + S3 tally +# scripts/sync_scratch_datasets_to_s3.sh dryrun [opts] # aws s3 sync --dryrun (lists, transfers nothing) +# scripts/sync_scratch_datasets_to_s3.sh cleanup [opts] # delete the sync pod + secret +# +# --only sync just one dataset dir (repeatable via comma list) +# --exclude pass through to `aws s3 sync --exclude` (repeatable) +# --wait after start, block and stream the log until the sync exits +# --keep (cleanup) leave the log dir on the PVC (default keeps it anyway) +# --profile local AWS profile to read creds from (default $AWS_PROFILE or 'op') +# --dst override destination prefix +# --concurrency aws s3 max_concurrent_requests inside the pod (default 64; zarr = many small files) +# --namespace/--pvc/--mount/--image cluster knobs (sane defaults below) +set -uo pipefail + +ACTION="start" +case "${1:-}" in start|status|dryrun|cleanup) ACTION="$1"; shift;; -*) : ;; "") : ;; *) echo "unknown action: $1"; exit 2;; esac + +PROFILE="${AWS_PROFILE:-op}" +NS="${SCRATCH_NS:-tower-nf}" +PVC="${SCRATCH_PVC:-tower-scratch}" +MOUNT="${SCRATCH_MOUNT:-/scratch}" +IMAGE="${SYNC_IMAGE:-amazon/aws-cli:latest}" +SRC_REL="task_ist_preprocessing/datasets" +DST="s3://openproblems-data/resources/task_ist_preprocessing/datasets/zarr3" +POD="${SYNC_POD:-s3sync-datasets}" +SECRET="${POD}-creds" +CONC=64 +WAIT=0 +ONLY="" +EXCLUDES=() + +while [ $# -gt 0 ]; do + case "$1" in + --only) ONLY="${ONLY:+$ONLY,}$2"; shift;; + --exclude) EXCLUDES+=( "$2" ); shift;; + --wait) WAIT=1;; + --keep) :;; + --profile) PROFILE="$2"; shift;; + --dst) DST="$2"; shift;; + --concurrency) CONC="$2"; shift;; + --namespace) NS="$2"; shift;; + --pvc) PVC="$2"; shift;; + --mount) MOUNT="$2"; shift;; + --image) IMAGE="$2"; shift;; + -h|--help) sed -n '2,50p' "$0"; exit 0;; + *) echo "unknown option: $1"; exit 2;; + esac + shift +done + +SRC="$MOUNT/$SRC_REL" +LOGDIR="$MOUNT/.s3sync" +LOG="$LOGDIR/${POD}.log" +DONE="$LOGDIR/${POD}.exit" + +command -v kubectl >/dev/null 2>&1 || { echo "ERROR: kubectl not found"; exit 2; } +command -v aws >/dev/null 2>&1 || { echo "ERROR: aws CLI not found (needed to read the '$PROFILE' creds)"; exit 2; } +kx(){ kubectl -n "$NS" "$@"; } + +# ---------- status ---------- +if [ "$ACTION" = "status" ]; then + echo "=== pod ==="; kx get pod "$POD" -o wide 2>&1 | grep -vE '^$' || true + if kx get pod "$POD" >/dev/null 2>&1; then + echo "=== sync log tail ==="; kx exec "$POD" -- sh -c "tail -n 25 '$LOG' 2>/dev/null || echo '(no log yet)'" + echo "=== exit marker ==="; kx exec "$POD" -- sh -c "[ -f '$DONE' ] && echo \"FINISHED rc=\$(cat '$DONE')\" || echo 'still running (no exit marker)'" + echo "=== objects under dst so far ==="; kx exec "$POD" -- sh -c "aws s3 ls --recursive '$DST/' 2>/dev/null | wc -l | xargs echo 'S3 objects:'" + fi + exit 0 +fi + +# ---------- cleanup ---------- +if [ "$ACTION" = "cleanup" ]; then + kx delete pod "$POD" --ignore-not-found --wait=false + kx delete secret "$SECRET" --ignore-not-found + echo "Deleted pod/$POD and secret/$SECRET (log kept at $LOG on the PVC)." + exit 0 +fi + +# ---------- start / dryrun: read creds, ensure secret + pod ---------- +AK="$(aws configure get aws_access_key_id --profile "$PROFILE" 2>/dev/null)" +SK="$(aws configure get aws_secret_access_key --profile "$PROFILE" 2>/dev/null)" +RG="$(aws configure get region --profile "$PROFILE" 2>/dev/null)"; RG="${RG:-us-west-2}" +[ -n "$AK" ] && [ -n "$SK" ] || { echo "ERROR: could not read aws creds for profile '$PROFILE'"; exit 3; } + +echo ">>> Ensuring k8s secret $SECRET (creds from local profile '$PROFILE') ..." +kx delete secret "$SECRET" --ignore-not-found >/dev/null 2>&1 +kx create secret generic "$SECRET" \ + --from-literal=AWS_ACCESS_KEY_ID="$AK" \ + --from-literal=AWS_SECRET_ACCESS_KEY="$SK" \ + --from-literal=AWS_DEFAULT_REGION="$RG" >/dev/null || { echo "ERROR: secret create failed"; exit 4; } + +if ! kx get pod "$POD" >/dev/null 2>&1; then + echo ">>> Spawning sync pod $POD (image $IMAGE, mounts $PVC at $MOUNT) ..." + cat </dev/null +apiVersion: v1 +kind: Pod +metadata: + name: $POD + namespace: $NS + labels: { app: s3sync-datasets } +spec: + restartPolicy: Never + containers: + - name: sync + image: $IMAGE + command: ["/bin/sh","-c","sleep infinity"] + envFrom: + - secretRef: { name: $SECRET } + volumeMounts: + - name: scratch + mountPath: $MOUNT + volumes: + - name: scratch + persistentVolumeClaim: { claimName: $PVC } +YAML + echo ">>> Waiting for pod Ready ..." + kx wait --for=condition=Ready "pod/$POD" --timeout=300s || { echo "ERROR: pod not Ready"; kx describe pod "$POD" | tail -20; exit 5; } +fi + +# preflight: creds + egress + dest reachable (fail fast, cheap) +echo ">>> Preflight: checking S3 write access to $DST ..." +kx exec "$POD" -- sh -c " + aws configure set default.s3.max_concurrent_requests $CONC + aws configure set default.s3.max_queue_size 10000 + echo ok | aws s3 cp - '$DST/.synctest' >/dev/null 2>&1 && aws s3 rm '$DST/.synctest' >/dev/null 2>&1 && echo ' write OK' || { echo ' ERROR: cannot write to $DST (creds/egress/bucket policy)'; exit 1; } +" || exit 6 + +# build the sync command (per-dataset loop for progress; resumable; additive) +EXC="" +for g in "${EXCLUDES[@]:-}"; do [ -n "$g" ] && EXC="$EXC --exclude '$g'"; done +DRY=""; [ "$ACTION" = "dryrun" ] && DRY="--dryrun" + +# which datasets +if [ -n "$ONLY" ]; then + LIST="$(echo "$ONLY" | tr ',' ' ')" +else + LIST="$(kx exec "$POD" -- sh -c "ls -1 '$SRC' 2>/dev/null")" +fi +[ -n "$LIST" ] || { echo "ERROR: no dataset dirs found under $SRC"; exit 7; } +N=$(echo "$LIST" | wc -w | tr -d ' ') +LIST="$(echo "$LIST" | tr '\n' ' ')" # space-separate: `for d in $LIST` must be one line in the runner +echo ">>> ${ACTION}: $N dataset dir(s)" +echo " $SRC/ -> $DST/" + +# Build the runner as a FILE on the PVC and launch from the file. Do NOT inline it +# into `nohup sh -c "