DynaCell v1.1 application

.github/workflows/test.yml

@@ -179,17 +179,37 @@ jobs:
         # segmenter + feature-extractor model loads — but pipeline import still
         # needs the eval stack). test_evaluation_grouped drives the
         # multi-condition driver against the same cache-only fixture.
+        # test_evaluation_microssim_calibration pins that MicroMS3IM calibration
+        # propagates OOM, drops constant GT slices from the fit pool, and scores
+        # NaN only on a degenerate GT pool (real cubic, CPU).
+        #
+        # test_submit_benchmark_batch covers the OTHER two srun emitters (array
+        # + chained-serial), which drifted out of sync with the single-job
+        # template because only the latter was under test. test_distribution,
+        # test_croissant and test_main_cli_routing cover the release path: the
+        # OZX packer / MANIFEST ledger, the per-dataset Croissant attribution,
+        # and the entry-point wiring that kept both of those CLIs reachable.
+        # All four are CPU-only and need no extra deps.
         run: |
           uv run --frozen --no-sync pytest \
             tests/test_benchmark_config_composition.py \
             tests/test_submit_benchmark_job.py \
+            tests/test_submit_benchmark_batch.py \
             tests/test_runtime.py \
             tests/test_evaluation_pipeline_parallel.py \
             tests/test_evaluation_pipeline_parallel_cpu.py \
             tests/test_evaluation_grouped.py \
+            tests/test_evaluation_microssim_calibration.py \
+            src/dynacell/evaluation/segmentation_test.py \
+            tests/test_distribution.py \
+            tests/test_croissant.py \
+            tests/test_main_cli_routing.py \
             tests/test_engine.py \
             tests/test_pipeline_cache.py \
+            tests/test_evaluation_fid_provenance.py \
             src/dynacell/evaluation/feature_metrics_test.py \
+            src/dynacell/evaluation/provenance_test.py \
+            tools/watch_stalled_jobs_test.py \
             -v
         working-directory: applications/dynacell
 

CLAUDE.md

@@ -55,6 +55,68 @@ When the user says "cancel all jobs," scope it to **batch jobs only**, never the
 
 **Subagent prompts for job status:** ask for completeness vs config, not just liveness. A prompt like "check the liveness of wandb run X" returns `state: finished` for a SIGTERM'd run and reads as success. Phrase it as "is run X complete relative to its configured `max_epochs`, and what was the exit reason (clean finish, scancel, OOM, timeout, exception)?"
 
+### Hung-but-allocated jobs
+
+**A job can finish its work and never exit.** Job `35083019_0` (a `pix2pix3d` predict) wrote its last chunk at 2026-07-30 19:20, then held an A6000 + 32 CPUs + 256 GB for **17.5 h** doing nothing. `squeue` shows `RUNNING` with a climbing wall clock — indistinguishable from healthy compute — and the rich progress bar only reaches `.out` at exit, so the log looks normal too. Measured rate: 1 in 509 allocations over 26 days.
+
+**The discriminator is CPU time, and only its derivative.** Across those 509 allocations every healthy job spent CPU at >= 0.93x wall; the hung one sat at 0.561 with a *zero* incremental rate. Absolute ratio alone is useless (a 16-CPU fit runs at ~7.8x wall), so compare two samples: `sstat -j <id> -a -P --format=JobID,AveCPU` — `-P` is mandatory, the default width truncates to `10-15:41:+` and misparses.
+
+**A full-looking output store plus a `RUNNING` job is NOT this signature.** On a
+re-predict launched with `--overwrite`, the writer rewrites chunks in place and the previous
+run's chunks stay on disk, so the store reads 100% complete from the first second — 36 of 36
+CellDiff-2D predicts looked finished-but-hung this way on 2026-09-11 while every one was
+healthy at a 1.01x incremental CPU rate. Judge on the CPU derivative above, never on "the
+output looks done"; see *Re-predict completeness* in `applications/dynacell/CLAUDE.md` for
+the mtime-based gate. In the same direction, a single `find -newermt '-20 minutes'` returning
+nothing is not evidence of a stall either: `/hpc` attribute caching hides fresh writes from a
+login node, and listing real mtimes (`find -printf '%T@ %p' | sort -rn`) showed writes
+seconds old at the same moment.
+
+**Using the stall watchdog.** `applications/dynacell/tools/watch_stalled_jobs.py` automates exactly that comparison. Start it whenever a campaign has long jobs in flight and leave it running:
+
+```sh
+# one-shot check: exit 0 = no stall detected, exit 1 = something is stalled, exit 2 = tool error
+uv run --no-sync python applications/dynacell/tools/watch_stalled_jobs.py --once
+
+# continuous, 10-min poll (launch in the background; it runs until killed)
+uv run --no-sync python applications/dynacell/tools/watch_stalled_jobs.py --interval 600
+```
+
+Quiet polls print `no stall detected, tracking N: <stepid>(<samples>) ...`, or `collecting history` when a step lacks a 15-minute baseline. A stalled step prints one `STALLED <stepid> <name> on <node>: wall Xh, cpu Yh, burned Z core-s/s over the last Wh` line per poll. Reading it:
+
+- It **only reports — it never cancels.** Cancelling a job with `afterok` dependents strands them, so follow the kill order above by hand.
+- **Interactive sessions are excluded by name** (`nomachine`, `gpu-hold`, `interactive`, bare `bash`/`sh`/`srun`). A renamed interactive session would get flagged — it still would not be cancelled, but don't act on the alert without checking.
+- It needs **two samples >= 15 min apart** and a step **>= 30 min old**, so expect no verdict on a fresh step for the first couple of polls.
+- Both modes persist samples in `$XDG_CACHE_HOME/viscy/watch_stalled_jobs-<user>@<hostname>.json` (default cache root: `~/.cache`), so each node keeps its own history by default; `--state-file` overrides the path. The file records the host that wrote it and refuses to load on any other, whatever its schema version: `~/.cache` is on NFS, where `flock` only excludes processes on the same node, so run the daemon and its `--once` checks on one node. A `--state-file` shared between nodes is claimed by the first host to create it (`O_EXCL`); a second host that started from the absent file fails on its first save instead of overwriting the winner's history. Repeat `--once` after at least 15 minutes. Its first invocation collects history; exit 0 does not establish that every job is healthy.
+- Histories are **per SLURM step** (`<jobid>.<step>` from `squeue -s`), judged against that step's own elapsed time rather than the job's. A `submit_benchmark_batch` chain advancing to its next `srun` step, or a requeue reusing a step id, starts a fresh history with that step's own clocks; the previous step's CPU cannot establish progress for the new one. A step first seen mid-allocation is judged on its cumulative CPU over its own age, so a watcher started late still catches a step that was already hung. Only `.extern` is skipped: `.batch` is tracked like any other step, because `submit_benchmark_batch --parallel` and `run_eval_direct.slurm` run their compute directly in the batch script with no `srun` step; for `srun`-based jobs the batch step never burns CPU, so it never qualifies.
+- A step that has not burned CPU **over a window of at least 30 minutes** is never flagged: startup NFS staging is legitimately ~0% CPU, and a short import-time burst must not qualify a step either. Qualification uses cumulative CPU over the step's age at the first sample taken once the step is >= 30 min old (whether or not that is the first sample), or CPU since first sight (only once >= 30 min have elapsed). A step that once qualified stays qualified through a long idle.
+- `--user` defaults to `alex.kalinin`; `sstat` only works on your own running jobs, so it cannot watch someone else's.
+
+**Predict wall limits are per family, sized from measurement** (`launcher_profiles/`):
+
+| Profile | `time` | Basis |
+|---|---|---|
+| `hardware_predict_any_gpu.yml` | 2 days | longest single-pass predict measured 21.9 h |
+| `hardware_predict_celldiff.yml` | 7 days | CELL-Diff runs 5.8-95.6 h; 8 predicts TIMEOUTed at the old 4-day cap on 2026-07-19 |
+| `hardware_h200_single.yml` | 4 days | **shared with 40 fit leaves** — do not re-tune from predict data |
+
+A new slow family gets its **own profile**; raising a shared cap to cover it makes the cap meaningless. `test_predict_leaf_wall_limit_matches_its_family` pins all 421 predict leaves to this table, and `generate_hek_predict_configs.py` picks the profile from `_HARDWARE_PROFILE` so regeneration cannot revert it.
+
+**Fit wall limits, same rule.** `hardware_4gpu.yml` is 4 days; `hardware_4gpu_long.yml` is 7 days and carries **only** the four joint ER/mito FCMAE leaves, which each TIMEOUTed twice at 4 days. Measured 1.58-1.82 epochs/h ⇒ 110-126 h for `max_epochs: 200`. Joint fits are the slow case because they pool iPSC + A549, roughly doubling steps per epoch. A 7-day 4-GPU H100/H200 request backfills materially worse, so `test_only_measured_slow_fits_get_the_long_wall` guards both directions — add a leaf only with a measured epochs/h rate.
+
+**Measure epochs/h from consecutive retained checkpoint mtimes**, not wall time ÷ steps. Each `epoch=N-step=M.ckpt` is an `(epoch, wall-clock)` pair, and two intervals inside one allocation cross-check each other (the ER-pretrained fit reproduced to three digits). Note `save_top_k` means the highest *retained* epoch is a best-by-monitor epoch, not progress — read `epoch` out of `last.ckpt` for that (`torch.load(..., mmap=True)`).
+
+**Two traps when killing one:**
+- **Lightning swallows SIGTERM.** `signal_connector.py` installs a handler that logs `Received SIGTERM` / `Bypassing SIGTERM` and sets a flag — it does not exit. `scancel` alone cannot stop a Lightning process outside its training loop; it dies on the KILL escalation (`ExitCode 0:9`), which took ~6 min here.
+- **Clear dependents' `Dependency=` BEFORE `scancel`**, or `afterok` chains land in `DependencyNeverSatisfied` permanently. `scontrol update JobId=<dep> Dependency=` to drop it, or `Dependency=afterok:<other>` to re-point the chain.
+
+**Forensics — capture this BEFORE cancelling, or the cause is unknowable:**
+1. `grep State /proc/<pid>/status` and `cat /proc/<pid>/task/*/wchan` (both readable without root, `ptrace_scope=0` here). `State: D` names an uninterruptible storage/driver call; `S` means a Python-level block.
+2. `py-spy dump --pid <pid>` (`uv tool install py-spy`) gives the full Python stack. If it errors `Failed to find python version from target process`, that is *itself* the answer: the process already reached interpreter finalization, so `trainer.predict()` returned and the writer's `plate.close()` completed.
+3. **Do not rely on SIGABRT + `PYTHONFAULTHANDLER=1`** — verified to produce no dump once the interpreter is finalizing.
+
+**Known unbounded wait in the predict teardown path:** zarr 3.2.1 `core/sync.py:89` registers `cleanup_resources` via `atexit`, which calls `_executor.shutdown(wait=True)` with no timeout — one stuck `zarr_pool` thread blocks interpreter exit forever at 0% CPU. (The same function caps its io-thread join at `timeout=0.2` "to avoid hanging"; the executor shutdown was left unbounded.) That is the proximate frame, but not the whole story here: a pure Python join still lets the SIGTERM handler run and log, and job `35083019_0` never logged it, so the terminal block was a C-level uninterruptible call underneath.
+
 ### Joint vs single-set training batch semantics
 
 `HCSDataModule` and `BatchedConcatDataModule` produce the same number of GPU samples per training step — but the YAML `batch_size` value that gets there is **different by a factor of `num_samples`**. Easy to misread either by skimming.

applications/dynacell/.gitignore

@@ -1,3 +1,6 @@
 lightning_logs/
 outputs/
 __pycache__/
+
+# Investigations, ablations and checks: one dated directory each, never tracked
+experiments/

applications/dynacell/CITATION.cff

@@ -0,0 +1,118 @@
+cff-version: 1.2.0
+title: "The DynaCell Evaluation Suite for Dynamic 3D Virtual Staining of Live Cells"
+message: >-
+  If you use DynaCell (code, data, or trained checkpoints), please cite the
+  paper listed under preferred-citation.
+type: software
+authors:
+  - given-names: Alexandr A.
+    family-names: Kalinin
+    affiliation: Biohub, San Francisco
+  - given-names: Dihan
+    family-names: Zheng
+    affiliation: University of California, San Francisco
+  - given-names: Taylla Milena
+    family-names: Theodoro
+    affiliation: Biohub, San Francisco
+  - given-names: Ivan
+    family-names: Ivanov
+    affiliation: Biohub, San Francisco
+  - given-names: Eduardo
+    family-names: Hirata-Miyasaki
+    affiliation: Biohub, San Francisco
+  - given-names: See-Chi
+    family-names: Lee
+    affiliation: Biohub, San Francisco
+  - given-names: Aofei
+    family-names: Liu
+    affiliation: Biohub, San Francisco
+  - given-names: Sricharan Reddy
+    family-names: Varra
+    affiliation: Biohub, San Francisco
+  - given-names: Talon
+    family-names: Chandler
+    affiliation: Biohub, San Francisco
+  - given-names: Soorya
+    family-names: Pradeep
+    affiliation: Biohub, San Francisco
+  - given-names: Chad
+    family-names: Liu
+    affiliation: Biohub, San Francisco
+  - given-names: Manuel D.
+    family-names: Leonetti
+    affiliation: Biohub, San Francisco
+  - given-names: Carolina
+    family-names: Arias
+    affiliation: Biohub, San Francisco
+  - given-names: Bo
+    family-names: Huang
+    affiliation: University of California, San Francisco
+  - given-names: Shalin B.
+    family-names: Mehta
+    affiliation: Biohub, San Francisco
+repository-code: "https://github.com/mehta-lab/VisCy"
+license: BSD-3-Clause
+abstract: >-
+  The DynaCell evaluation suite assesses 3D time-lapse virtual staining of live
+  cells. It combines paired 3D time-lapse images acquired with and without
+  fluorescent labels, a set of baseline models, and three metric tiers that
+  measure pixel-level fidelity, organelle segmentation performance, and
+  single-cell phenotype similarity. This evaluation suite is part of DynaCell,
+  a broader framework for image-based profiling of cell-state dynamics.
+keywords:
+  - virtual staining
+  - live-cell imaging
+  - benchmarking
+  - microscopy
+  - deep learning
+preferred-citation:
+  type: conference-paper
+  title: "The DynaCell Evaluation Suite for Dynamic 3D Virtual Staining of Live Cells"
+  collection-title: "Advances in Neural Information Processing Systems (NeurIPS 2026), Evaluations and Datasets Track"
+  year: 2026
+  authors:
+    - given-names: Alexandr A.
+      family-names: Kalinin
+      affiliation: Biohub, San Francisco
+    - given-names: Dihan
+      family-names: Zheng
+      affiliation: University of California, San Francisco
+    - given-names: Taylla Milena
+      family-names: Theodoro
+      affiliation: Biohub, San Francisco
+    - given-names: Ivan
+      family-names: Ivanov
+      affiliation: Biohub, San Francisco
+    - given-names: Eduardo
+      family-names: Hirata-Miyasaki
+      affiliation: Biohub, San Francisco
+    - given-names: See-Chi
+      family-names: Lee
+      affiliation: Biohub, San Francisco
+    - given-names: Aofei
+      family-names: Liu
+      affiliation: Biohub, San Francisco
+    - given-names: Sricharan Reddy
+      family-names: Varra
+      affiliation: Biohub, San Francisco
+    - given-names: Talon
+      family-names: Chandler
+      affiliation: Biohub, San Francisco
+    - given-names: Soorya
+      family-names: Pradeep
+      affiliation: Biohub, San Francisco
+    - given-names: Chad
+      family-names: Liu
+      affiliation: Biohub, San Francisco
+    - given-names: Manuel D.
+      family-names: Leonetti
+      affiliation: Biohub, San Francisco
+    - given-names: Carolina
+      family-names: Arias
+      affiliation: Biohub, San Francisco
+    - given-names: Bo
+      family-names: Huang
+      affiliation: University of California, San Francisco
+    - given-names: Shalin B.
+      family-names: Mehta
+      affiliation: Biohub, San Francisco