Rescale and LOT correction

.claude/skills/airtable-build-collection/SKILL.md

@@ -32,6 +32,7 @@ Key fields in the Datasets table:
 | `channel_0_marker` .. `channel_N_marker` | Protein marker for each channel |
 | `t_shape`, `c_shape`, `z_shape`, `y_shape`, `x_shape` | Array dimensions |
 | `pixel_size_xy_um`, `pixel_size_z_um` | Physical pixel sizes |
+| `microscope` | Acquisition platform (e.g. `mantis_v1`, `mantis_v2`); derive from pixel size if absent |
 
 ## Usage
 
@@ -155,13 +156,18 @@ experiments:
     moi: <moi or 0.0>
     pixel_size_xy_um: <from airtable>
     pixel_size_z_um: <from airtable>
+    microscope: <from airtable microscope field; else derive: mantis_v1 / mantis_v2>
 ```
 
 Key notes:
 - `data_path` should be the zarr store root (up to `.zarr`), NOT the FOV-level path from Airtable
 - `perturbation_wells` uses `uninfected` / `<perturbation>` keys inferred from the `perturbation` field
 - `channels` lists only the channels needed for training (not all channels in the zarr)
 - `marker` at the experiment level is the primary marker for this experiment entry
+- `microscope` labels the acquisition platform — always populate it. Use the Airtable microscope
+  field when present; otherwise derive from acquisition grid: `pixel_size_xy_um == 0.1494` →
+  `mantis_v1`, `0.1133` → `mantis_v2`. Downstream QC (embedding-consistency-qc) partitions
+  datasets by `obs["microscope"]`, so a missing/empty value silently collapses the v1/v2 split.
 
 ### Step 7: Save and Validate
 

.gitignore

@@ -20,6 +20,8 @@ __pycache__/
 
 # Virtual environments
 .venv/
+.venv-phenotyping/
+.phenotyping-batch/
 venv/
 ENV/
 
@@ -73,6 +75,3 @@ lightning_logs/
 checkpoints/
 
 plot_related/
-
-# Local-only planning docs (not for upstream)
-applications/dynaclr/docs/DAGs/evaluation_matrix.md

applications/dynaclr/README.md

@@ -48,6 +48,18 @@ uv run --package dynaclr viscy predict -c examples/configs/predict.yml
 sbatch examples/configs/fit_slurm.sh
 ```
 
+### End-to-end pipeline (new dataset → embeddings → eval)
+
+For the full "dataset → per-marker embeddings → downstream evals" workflow — including the
+one-call `dynaclr predict-triplet` (writes the dataset-centric tree
+`<dataset>/2-phenotyping/predictions/{model}/{run}/{ckpt}/{marker}.zarr`), the decoupled
+Nextflow `eval_from_embeddings` entry, and the **model matrix** for running many models through
+train → predict → eval in parallel — see:
+
+- [`docs/DAGs/end_to_end.md`](docs/DAGs/end_to_end.md) — the pipeline overview + Quickstart runbook
+- [`nextflow/README.md`](nextflow/README.md) — the two Nextflow eval entries
+- [`tools/README.md`](tools/README.md) — the model matrix (`dynaclr run-matrix`) + launchers
+
 The YAML config determines which model and data module to use via `class_path`:
 
 ```yaml
@@ -78,12 +90,21 @@ DynaCLR also provides evaluation-specific commands via `dynaclr <command>`:
 
 | Command | Description |
 |---------|-------------|
+| `predict-triplet` | One-call per-marker embedding inference from a collection + checkpoint → dataset-centric tree |
+| `predict-batch` | Run `predict-triplet` for one model with AI-readiness preflight |
+| `eval` | Evaluate frozen embeddings selected by model/run/checkpoint |
+| `run-matrix` | Chain train → predict → eval for many models through SLURM dependencies |
+| `split-embeddings` | Split a combined embeddings zarr per experiment/marker (`--route-by-dataset` → the dataset tree) |
+| `embedding-consistency-qc` | Build per-marker cross-dataset MMD² and correlation matrices |
+| `run-linear-classifiers` | Train linear classifiers on embeddings (batch, CSV metrics) |
+| `append-annotations` / `append-predictions` | Join annotation / predicted-label columns onto per-experiment zarrs |
 | `train-linear-classifier` | Train a linear classifier on cell embeddings |
 | `apply-linear-classifier` | Apply a trained linear classifier to new embeddings |
 | `append-obs` | Append columns from a CSV to an AnnData zarr obs (with optional prefix, e.g. `annotated_`, `feature_`) |
 | `reduce-dimensionality` | Compute PCA, UMAP, and/or PHATE on saved embeddings |
 | `evaluate-smoothness` | Evaluate temporal smoothness of embedding models |
 | `compare-models` | Compare previously saved smoothness results |
+| `compute-mmd` | MMD between perturbation / experiment groups (cross-experiment consistency) |
 | `info` | Print summary of an AnnData zarr store |
 
 ```bash

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2024_11_07_A549_SEC61_DENV.yml

@@ -0,0 +1,41 @@
+name: organelle-box-2024_11_07_A549_SEC61_DENV
+description: "Profiling/handoff collection: 2024_11_07 A549 SEC61 DENV (single-marker
+  organelle box). Channels embedded as bag-of-channels samples: SEC61B (GFP),
+  viral_sensor (mCherry), Phase3D. Prepared zarr on VAST with timepoint_statistics +
+  focus_slice. Sourced from Airtable base app8vqaoWyOwa0sB5 (dataset
+  '2024_11_07_A549_SEC61_DENV'). FOV C/1/000000 exists in the zarr but has no tracking
+  CSV and is excluded at build time."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "{dataset}=\"2024_11_07_A549_SEC61_DENV\""
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2024_11_07_A549_SEC61_DENV
+    data_path: ${datasets_root}/2024_11_07_A549_SEC61_DENV/2024_11_07_A549_SEC61_DENV.zarr
+    tracks_path: ${datasets_root}/2024_11_07_A549_SEC61_DENV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: SEC61B
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - B/1
+        - B/3
+      DENV:
+        - B/2
+        - C/2
+    exclude_fovs:
+      - C/1/000000
+    interval_minutes: 10.0
+    start_hpi: 4.0
+    marker: SEC61B
+    organelle: endoplasmic_reticulum
+    moi: 5.0
+    pixel_size_xy_um: 0.1494
+    pixel_size_z_um: 0.174

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_08_A549_G3BP1_ZIKV.yml

@@ -0,0 +1,38 @@
+name: organelle-box-2026_04_08_A549_G3BP1_ZIKV
+description: "Embed-only collection: 2026_04_08 A549 G3BP1 ZIKV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: G3BP1 (GFP), viral_sensor (mCherry), Phase3D.
+  Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_08_A549_G3BP1_ZIKV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_08_A549_G3BP1_ZIKV
+    data_path: ${datasets_root}/2026_04_08_A549_G3BP1_ZIKV/2026_04_08_A549_G3BP1_ZIKV.zarr
+    tracks_path: ${datasets_root}/2026_04_08_A549_G3BP1_ZIKV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: G3BP1
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - D/6
+      ZIKV:
+        - B/5
+        - B/6
+        - C/5
+        - C/6
+        - D/5
+    interval_minutes: 30.0
+    start_hpi: 3.0
+    marker: G3BP1
+    organelle: stress_granules
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_10_A549_TOMM20_ZIKV.yml

@@ -0,0 +1,38 @@
+name: organelle-box-2026_04_10_A549_TOMM20_ZIKV
+description: "Embed-only collection: 2026_04_10 A549 TOMM20 ZIKV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: TOMM20 (GFP), viral_sensor (mCherry), Phase3D.
+  Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_10_A549_TOMM20_ZIKV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_10_A549_TOMM20_ZIKV
+    data_path: ${datasets_root}/2026_04_10_A549_TOMM20_ZIKV/2026_04_10_A549_TOMM20_ZIKV.zarr
+    tracks_path: ${datasets_root}/2026_04_10_A549_TOMM20_ZIKV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: TOMM20
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - A/2
+      ZIKV:
+        - A/3
+        - A/4
+        - B/2
+        - B/3
+        - B/4
+    interval_minutes: 30.0
+    start_hpi: 5.0
+    marker: TOMM20
+    organelle: mitochondria
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_14_A549_SEC61B_DENV.yml

@@ -0,0 +1,38 @@
+name: organelle-box-2026_04_14_A549_SEC61B_DENV
+description: "Embed-only collection: 2026_04_14 A549 SEC61B DENV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: SEC61B (GFP), viral_sensor (mCherry), Phase3D.
+  Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_14_A549_SEC61B_DENV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_14_A549_SEC61B_DENV
+    data_path: ${datasets_root}/2026_04_14_A549_SEC61B_DENV/2026_04_14_A549_SEC61B_DENV.zarr
+    tracks_path: ${datasets_root}/2026_04_14_A549_SEC61B_DENV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: SEC61B
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - C/2
+      DENV:
+        - C/3
+        - C/4
+        - D/2
+        - D/3
+        - D/4
+    interval_minutes: 30.0
+    start_hpi: 3.0
+    marker: SEC61B
+    organelle: endoplasmic_reticulum
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_21_A549_G3BP1_DENV.yml

@@ -0,0 +1,41 @@
+name: organelle-box-2026_04_21_A549_G3BP1_DENV
+description: "Embed-only collection: 2026_04_21 A549 G3BP1 DENV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: G3BP1 (GFP), viral_sensor (mCherry), Phase3D.
+  Source zarrs are the intracellular_dashboard preprocess outputs (5-assemble data +
+  4-track ultrack labels); no prepared VAST copy exists. Predict uses fov_statistics
+  normalization, so timepoint_statistics is not required. Annotations exist (infection,
+  organelle, cell_division, cell_death)."
+datasets_root: /hpc/projects/intracellular_dashboard/organelle_dynamics
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_21_A549_G3BP1_DENV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_21_A549_G3BP1_DENV
+    data_path: ${datasets_root}/2026_04_21_A549_G3BP1_DENV/1-preprocess/5-assemble/2026_04_21_G3BP1_DENV.zarr
+    tracks_path: ${datasets_root}/2026_04_21_A549_G3BP1_DENV/1-preprocess/4-track/2026_04_21_G3BP1_DENV.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: G3BP1
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - A/1
+      DENV:
+        - A/2
+        - A/3
+        - B/1
+        - B/2
+        - B/3
+    interval_minutes: 30.0
+    start_hpi: 3.0
+    marker: G3BP1
+    organelle: stress_granules
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_28_A549_SEC61B_DENV.yml

@@ -0,0 +1,38 @@
+name: organelle-box-2026_04_28_A549_SEC61B_DENV
+description: "Embed-only collection: 2026_04_28 A549 SEC61B DENV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: SEC61B (GFP), viral_sensor (mCherry), Phase3D.
+  Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_28_A549_SEC61B_DENV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_28_A549_SEC61B_DENV
+    data_path: ${datasets_root}/2026_04_28_A549_SEC61B_DENV/2026_04_28_A549_SEC61B_DENV.zarr
+    tracks_path: ${datasets_root}/2026_04_28_A549_SEC61B_DENV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: SEC61B
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - A/2
+      DENV:
+        - A/3
+        - A/4
+        - B/2
+        - B/3
+        - B/4
+    interval_minutes: 30.0
+    start_hpi: 3.0
+    marker: SEC61B
+    organelle: endoplasmic_reticulum
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995

applications/dynaclr/configs/collections/organelle-box-denv-zikv/2026_04_30_A549_TOMM20_DENV.yml

@@ -0,0 +1,38 @@
+name: organelle-box-2026_04_30_A549_TOMM20_DENV
+description: "Embed-only collection: 2026_04_30 A549 TOMM20 DENV (single-marker organelle box).
+  Channels embedded as bag-of-channels samples: TOMM20 (GFP), viral_sensor (mCherry), Phase3D.
+  Prepared zarr on VAST with timepoint_statistics + focus_slice. No annotations."
+datasets_root: /hpc/projects/organelle_phenotyping/datasets
+
+provenance:
+  airtable_base_id: app8vqaoWyOwa0sB5
+  airtable_query: "SEARCH(\"2026_04_30_A549_TOMM20_DENV\", {dataset})"
+  created_by: "eduardo.hirata"
+
+experiments:
+  - name: 2026_04_30_A549_TOMM20_DENV
+    data_path: ${datasets_root}/2026_04_30_A549_TOMM20_DENV/2026_04_30_A549_TOMM20_DENV.zarr
+    tracks_path: ${datasets_root}/2026_04_30_A549_TOMM20_DENV/tracking.zarr
+    channels:
+      - name: raw GFP EX488 EM525-45
+        marker: TOMM20
+      - name: raw mCherry EX561 EM600-37
+        marker: viral_sensor
+      - name: Phase3D
+        marker: Phase3D
+    perturbation_wells:
+      uninfected:
+        - A/2
+      DENV:
+        - A/3
+        - A/4
+        - B/2
+        - B/3
+        - B/4
+    interval_minutes: 30.0
+    start_hpi: 4.0
+    marker: TOMM20
+    organelle: mitochondria
+    moi: 5.0
+    pixel_size_xy_um: 0.1133
+    pixel_size_z_um: 0.16995