Port shared VisCy package changes from DynaCell v1.1

.github/workflows/test.yml

@@ -91,6 +91,33 @@ jobs:
         run: uv run --frozen pytest --cov=viscy_data --cov-report=term-missing
         working-directory: packages/viscy-data
 
+  test-utils:
+    name: Test Utils (Python ${{ matrix.python-version }}, ubuntu-latest)
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: true
+      matrix:
+        python-version: ["3.12", "3.13"]
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+
+      - name: Set up uv with Python ${{ matrix.python-version }}
+        uses: astral-sh/setup-uv@v7
+        with:
+          python-version: ${{ matrix.python-version }}
+          enable-cache: true
+          cache-suffix: ubuntu-latest-${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: uv sync --frozen --all-extras --dev
+        working-directory: packages/viscy-utils
+
+      - name: Run tests with coverage
+        run: uv run --frozen pytest --cov=viscy_utils --cov-report=term-missing
+        working-directory: packages/viscy-utils
+
   test-applications:
     name: Test (${{ matrix.application }}, Python 3.13, ubuntu-latest)
     runs-on: ubuntu-latest
@@ -119,7 +146,7 @@ jobs:
         working-directory: applications/${{ matrix.application }}
 
   test-dynacell-configs:
-    name: Test dynacell benchmark configs (Python 3.13, ubuntu-latest)
+    name: Test dynacell configs and regressions (Python 3.13, ubuntu-latest)
     runs-on: ubuntu-latest
 
     steps:
@@ -139,10 +166,10 @@ jobs:
         # (falls back to numpy/scikit-image without CUDA), so the eval extra
         # installs and runs on a GPU-less runner. eval_gpu (cupy/cucim) is NOT
         # installed — those are the only CUDA-only deps.
-        run: uv sync --frozen --extra eval --group test
+        run: uv sync --frozen --package dynacell --extra eval --group test
         working-directory: applications/dynacell
 
-      - name: Run benchmark-schema + submit-tool + eval-runtime tests
+      - name: Run benchmark configs and pipeline regressions
         # test_benchmark_config_composition + test_submit_benchmark_job cover
         # config composition + the launcher. test_runtime + the two
         # test_evaluation_pipeline_parallel* suites cover the runtime module,
@@ -153,20 +180,23 @@ jobs:
         # needs the eval stack). test_evaluation_grouped drives the
         # multi-condition driver against the same cache-only fixture.
         run: |
-          uv run --frozen pytest \
+          uv run --frozen --no-sync pytest \
             tests/test_benchmark_config_composition.py \
             tests/test_submit_benchmark_job.py \
             tests/test_runtime.py \
             tests/test_evaluation_pipeline_parallel.py \
             tests/test_evaluation_pipeline_parallel_cpu.py \
             tests/test_evaluation_grouped.py \
+            tests/test_engine.py \
+            tests/test_pipeline_cache.py \
+            src/dynacell/evaluation/feature_metrics_test.py \
             -v
         working-directory: applications/dynacell
 
   check:
     name: All tests pass
     if: always()
-    needs: [test, test-data, test-data-extras, test-applications, test-dynacell-configs]
+    needs: [test, test-data, test-data-extras, test-utils, test-applications, test-dynacell-configs]
     runs-on: ubuntu-latest
     steps:
       - name: Verify all test jobs succeeded

CLAUDE.md

@@ -4,19 +4,7 @@
 
 VisCy is a **uv workspace monorepo** for virtual staining and computational microscopy. Sub-packages live under `packages/`.
 
-## Repo Layout
-
-```
-pyproject.toml              # Root config (ruff, pytest, uv workspace)
-packages/
-  viscy-data/               # Data loading and Lightning DataModules
-  viscy-models/             # Neural network architectures
-  viscy-transforms/         # Image transforms
-src/viscy/                  # Umbrella package (re-exports)
-applications/               # Self-contained research applications
-```
-
-### Packages vs Applications
+## Packages vs Applications
 
 - **Shared code belongs in `packages/`**, not in applications.
 - **Applications must not import from each other.** If two applications need the same logic, move it to an existing package or create a new one.
@@ -30,16 +18,6 @@ applications/               # Self-contained research applications
 
 Use `uv` package manager. Run commands with `uv run <command>`. Edit `pyproject.toml` to modify dependencies and sync to update `uv.lock`.
 
-```sh
-uv venv -p 3.13
-uv sync --all-packages --all-extras
-```
-
-If `uv` is not installed:
-```sh
-curl -LsSf https://astral.sh/uv/install.sh | sh
-```
-
 On HPC, symlink the uv cache out of your home directory first:
 ```sh
 mkdir -p /hpc/mydata/firstname.lastname/.cache/uv && ln -s /hpc/mydata/firstname.lastname/.cache/uv ~/.cache/uv
@@ -98,25 +76,12 @@ Examples (verified against the `applications/dynacell/configs/benchmarks/virtual
 
 When in doubt, read both `train_dataloader` overrides directly — they are short. Don't infer from comments alone.
 
-### Common Commands
-
-```sh
-uvx ruff check packages/        # lint
-uvx ruff check --fix packages/  # lint + auto-fix
-uvx ruff format packages/       # format
-uv run pytest                    # all tests
-```
-
 ### Testing
 
-```sh
-uv run pytest                          # all tests
-uv run pytest packages/viscy-data/     # single package (data)
-uv run pytest packages/viscy-models/   # single package (models)
-```
-
 Prefer `{file}_test.py` in the same directory as `{file}.py`, unless there are import issues, in which case use `tests/`.
 
+Run the suite in parallel with `pytest -n <cores>` (pytest-xdist, in the `test` group). The root `conftest.py` pins each worker to one OpenMP/BLAS thread. Without that pin, torch in every worker grabs every core and the parallel run is 10x slower than it should be. `-n auto` counts the cores the shell may use, so inside a 1-CPU `srun` step it runs a single worker; give the step more CPUs (`srun --overlap -c N`) first.
+
 ---
 
 ## Project Conventions
@@ -137,10 +102,6 @@ Prefer `{file}_test.py` in the same directory as `{file}.py`, unless there are i
 
 ### Code Style
 
-- Docstrings use **numpy style** (`convention = "numpy"`).
-- Lint rules: `D, E, F, I, NPY, PD, W`.
-- `D` rules are ignored in `**/tests/**` and notebooks.
-- Format: double quotes, spaces, 120 char line length.
 - Use a subagent to run tests and complex bash commands, especially those expected to return complex output.
 - Run independent tasks (multi-file edits across separate concerns, cross-cutting verifications, distinct review angles) in parallel via concurrent subagents in a single message. Subagent startup overhead is negligible relative to sequential blocking. Only sequence subagents when a later task needs an earlier task's output.
 
@@ -187,17 +148,7 @@ Ask yourself if your test is actually covering the true function.
 
 ### Coding Philosophy
 
-#### 1. Think Before Coding
-
-Don't assume. Don't hide confusion. Surface tradeoffs.
-
-Before implementing:
-- State your assumptions explicitly. If uncertain, ask.
-- If multiple interpretations exist, present them — don't pick silently.
-- If a simpler approach exists, say so. Push back when warranted.
-- If something is unclear, stop. Name what's confusing. Ask.
-
-#### 2. Simplicity First
+#### 1. Simplicity First
 
 Minimum code that solves the problem. Nothing speculative.
 
@@ -208,7 +159,7 @@ Minimum code that solves the problem. Nothing speculative.
 - If you write 200 lines and it could be 50, rewrite it.
 - Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.
 
-#### 3. Surgical Changes
+#### 2. Surgical Changes
 
 Touch only what you must. Clean up only your own mess.
 
@@ -223,18 +174,3 @@ When your changes create orphans:
 - Don't remove pre-existing dead code unless asked.
 
 The test: every changed line should trace directly to the user's request.
-
-#### 4. Goal-Driven Execution
-
-Define success criteria. Loop until verified.
-
-Transform tasks into verifiable goals:
-- "Add validation" → "Write tests for invalid inputs, then make them pass"
-- "Fix the bug" → "Write a test that reproduces it, then make it pass"
-- "Refactor X" → "Ensure tests pass before and after"
-
-For multi-step tasks, state a brief plan:
-1. [Step] → verify: [check]
-2. [Step] → verify: [check]
-
-Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.

applications/cytoland/examples/configs/recipes/modes/spotlight.yml

@@ -1,9 +1,10 @@
 # Mode recipe: Spotlight foreground-aware loss (Kalinin et al. 2025).
 # Cross-cutting: sets loss, fg_mask_key, nonzero filtering, AND
-# Otsu-centered normalization.  fg_threshold: 0.0 is valid ONLY
-# because normalizations use subtrahend: otsu_threshold.
+# Otsu-centered normalization. The Otsu-centred target is what makes 0 the
+# FG/BG boundary in the model's output space -- which the Dice term needs
+# regardless of fg_threshold, since that field is inert here (see below).
 #
-# Requires: viscy preprocess --compute_otsu --compute_fg_masks
+# Requires: viscy preprocess --compute_otsu true --compute_fg_masks true
 #
 # Base ordering: this mode must come AFTER the data recipe in the
 # base: list so that its normalizations replace the data default.
@@ -14,6 +15,10 @@ model:
       init_args:
         lambda_mse: 0.5
         sigmoid_k: -0.95
+        # INERT while data.fg_mask_key is set below: forward() short-circuits
+        # on `if fg_mask is not None`, so the mask from the store defines
+        # foreground and this threshold is never consulted. It applies only if
+        # fg_mask_key is removed. Same for data.nonzero_threshold.
         fg_threshold: 0.0
 data:
   init_args:

applications/cytoland/examples/configs/vscyto3d/train_spotlight.yml

@@ -1,5 +1,5 @@
 # VSCyto3D: supervised training with Spotlight foreground-aware loss.
-# Requires: viscy preprocess --compute_otsu --compute_fg_masks
+# Requires: viscy preprocess --compute_otsu true --compute_fg_masks true
 # Usage: python -m cytoland fit --config vscyto3d/train_spotlight.yml
 base:
   - ../recipes/trainer/fit.yml