DynaCell v1.1 by alxndrkalinin · Pull Request #494 · Biohub/VisCy

alxndrkalinin · 2026-07-29T20:53:25Z

No description provided.

The write gate compared cached and freshly recomputed PCC row by row at rtol=1e-3. That is the wrong scale for PCC: its numerator is a difference of nearly-cancelling sums, so on a near-degenerate prediction float32 cancellation alone moves a row by ~1e-3. FNet3D on HEK mitochondria sits at PCC 0.003 -- no correlation at all -- where 1e-3 is 30% of the value. 19 of 648 conditions in the 2026-08-03 pass tripped the per-row check while their means agreed to four decimals. Three were cross-checked against independent ground truth and all three showed the arrays were unchanged: pix2pix3d mito iPSC-trained -> A549 mock reproduced the published 0.540 as 0.5396, and the two HEK cells agreed to 8.6e-4 and 1.2e-2 against the pre-campaign CSV snapshot. Their prediction and GT stores are both provably older than the evals and untouched since, so there was no data change to catch. Gate on the mean instead, at 5e-4 -- tighter than the 3-decimal rounding the tables publish, so a mean inside it cannot change a cell, while a real re-predict moves it well outside. A per-row failure that clears the mean is reported as reproduced_mean rather than silently collapsed into reproduced, so the audit still surfaces it. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Writes down the trap that nearly cost GPU-days on 2026-08-03. PCC is the right signal for "did this cache come from the arrays on disk now" because it is affine-invariant, but a per-row relative tolerance is the wrong test for it: its numerator is a difference of nearly-cancelling sums, so on a near-degenerate prediction (FNet3D on HEK mito sits at PCC 0.003) float32 cancellation alone moves a row by 30%. rtol=1e-3 per row flagged 19 of 648 conditions as stale and every one that fed a table was a false positive. Also records the four checks that proved them false -- comparing the mean against an independently published number, full-scan chunk mtimes on both prediction and GT, checking for a live writer, and git log on the config surface -- and the rule they add up to: a staleness verdict is a hypothesis, so verify it before spending compute. A gate that fails closed is safe to leave over-sensitive; a false positive must never drive a re-run. Notes in passing that focus.* / slice_selection do not affect pixel metrics, since reaching for them to explain a moved pixel number was one of the wrong turns taken. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

W7a of the brightfield-input ablation, and a hard prerequisite for writing any *__bf* prediction to disk. discover_predictions calls parse_zarr_name unguarded, so an unregistered *__bf* store raises ValueError and breaks grouped-eval regeneration for EVERY bucket, not just the brightfield ones. ipsc__bf -> ipsc_bf_trained and a549__bf -> a549_bf_trained, each its own bucket. The a549__deconv -> a549_trained entry is a misleading precedent: deconv provenance marks how the target was produced from the same Phase3D input, so folding it groups like with like. __bf swaps the input channel itself -- raw Brightfield instead of the waveorder reconstruction derived from it -- so folding would collide a brightfield and a phase prediction on one canonical_identity and let one silently shadow the other. test_walk_predictions_yields_known_buckets asserted equality against a hardcoded 4x3. The brightfield arms cover nucleus and ER only and their predictions land incrementally, so equality would fail throughout the predict drain without ever indicating a problem. Reformulated as "the core grid is present" plus "no bucket label outside the table" -- strictly stronger, since it also catches an unregistered label the old check would have accepted had it been in the hardcoded set. test_parse_zarr_name_out_of_scope_train_set_raises used a549__bf as its out-of-scope example, which this commit makes in-scope. Repointed at a549__bf__deconv, which paths.py documents as a grammar-ready follow-up with no fits behind it. Deliberately NOT regenerating the grouped leaves, which the W7a sketch called for: the prediction tree has grown since they were last generated, so a blanket regeneration folds the 2D in-focus track into all 12 3D buckets -- scoring per-z 2D models on full-volume metrics they never targeted -- and flips force_recompute.final_metrics to true, defeating cache-skip. Verified by regenerating, diffing (2246 insertions across 14 files) and reverting to byte-identical. Brightfield eval will get filtered sibling buckets instead, which is an eval-time concern and no longer blocks the predicts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

W7b. Four predict sets per arm (own domain plus the three A549 conditions) across 2 train sets x 2 organelles x 2 models, so the brightfield arms get the same 2x2 cross-domain grid the phase models have. Every field is copied from the arm's own PHASE baseline leaf and verified against it: each brightfield leaf's fully composed config differs from its baseline in exactly 9 fields, all intended (source_channel, the normalization key, ckpt_path, output_store, run_root, job_name, experiment_id, trained_on, dataset_ref). A paired ablation must not also vary the predict recipe, so this is asserted rather than assumed. Checkpoint selection follows each model's own fit configuration: * fnet3d arms take the loss/validate-monitored callback's best_model_path, read out of last.ckpt. That file carries TWO ModelCheckpoint callbacks and the unmonitored one's best_model_path is just the final step (latest-epoch=N-step=200000) with best_model_score None -- taking the last callback found silently picks it. The monitored bests are nucleus 0.7935/0.1787 and ER 0.5222/0.5071 for iPSC/A549. * celldiff arms are save_top_k=-1 with no monitor, so the final epoch (epoch=19) is pinned explicitly. Verified that dataset_ref.source_channel resolves to Brightfield in the composed config for all four corners of the grid rather than being dropped -- DatasetRef sets extra="forbid" precisely because a silently ignored source_channel would train and predict on Phase3D while every log claimed brightfield, and a no-op ablation is worse than a failed one. Also confirmed the A549 nucleus leaves still resolve their gene-keyed h2b target and the iPSC-test leaves reach test_cropped. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Each __2d bucket goes from 12 to 16 conditions, adding celldiff_2d x {ipsc, a549 mock/denv/zikv} -- 48 new conditions in total. celldiff_2d was held out of these leaves on purpose: the generator's only freshness gate is pred_path.is_dir(), which passes on a partially-written store (4813 of 5787 chunk files, measured on a live one). Its A549-plate predicts have now finished, so the exclusion is retired. The gate applied by hand in place of the generator's: - sacct: all 48 CELLDiff2D_PRED jobs COMPLETED, ExitCode 0:0. - Per-store chunk counts for all 48 celldiff_2d prediction stores match the same organelle/pool/bucket fnet2d store exactly. The complete size is a property of (organelle, bucket) and not a constant: 4270-4300 on the iPSC buckets, 5211-5787 on the A549 ones. - er/a549__denv is a low outlier at 5211 in every pool, so it was cross-checked against two further fully-evaluated models -- both FCMAE-2D arms, also 5211 over the same 12 FOVs. That closes the one blind spot of an fnet2d-relative gate, which by construction cannot see a shortfall it shares with its own reference. Blocks were lifted verbatim from a generate_grouped_eval_configs.py --out-root run into scratch and inserted immediately before each test set's vscyto2d block, which reproduces the generator's own within-test-set ordering (celldiff_2d, vscyto2d, unext2_2d, fnet2d). The integrity gate is the load-bearing part: after inserting, deleting the 4 new blocks again had to restore every file byte for byte, which passed on all 12 and proves the already-evaluated conditions were not disturbed. Across the 12 buckets the 192 conditions carry 192 distinct pred_cache_dir and 192 distinct save_dir, and all 64 distinct pred_paths exist on disk. force_recompute.final_metrics stays false, against the generator's default of true, so the 12 pre-existing conditions cache-skip -- fnet2d and both FCMAE-2D arms have pixel and feature metrics in all 48 of their save dirs -- and only the 4 celldiff_2d conditions pay the cold deep-feature extraction. That extraction still needs --constraint a100_80|h100_80|h200; bare a100 matches the 40 GB nodes and OOMs. Headers updated in the same pass: scope 12 -> 16, the exclusion paragraph replaced with the gate above, and the cache-skip note corrected to name the three arms that are now evaluated rather than only fnet2d. Tests: generate_grouped_eval_configs_test.py 41/41; test_benchmark_config_composition.py + test_paths.py 1786 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KWqdbgQhTbBqxnwPBzKHhp

PatchGAN3D layers 1-4 used a cubic kernel_size=4 with padding=1, so a size-1 Z axis pads to 3 against a size-4 kernel and torch raises "Kernel size can't be greater than actual input size" at layer 1. Only layer 5 had been written Z-safe as (1, 4, 4)/(0, 1, 1). That made the discriminator the sole blocker for running pix2pix Z-preserving: the UNetViT generator already builds at [1, 512, 512] with a (1, 4, 4) patch, because it shares ViTBottleneck3D with CellDiff and 0126fe6 gave that per-axis patch extents. preserve_z collapses the kernel and padding on Z for layers 1-4 and drops the Z stride of layers 3-4, so Z passes through untouched rather than merely being tolerated -- a 5-plane slab comes out 5 planes, where the strided 3D topology would halve it twice. MultiScalePatchGAN3D forwards the flag; its inter-scale pooling was already YX-only. Verified: at Z=1 the two scales emit (1,1,1,31,31) and (1,1,1,15,15), the same YX patch grid the 3D config produces, and every conv weight receives a gradient. preserve_z=False reproduces the published 3D topology bit-for-bit -- same kernels, strides and forward output -- which matters because these are the convs all 14 pix2pix3d leaves train. 11/11 tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Fifth model in the 2D-vs-3D benchmark track, after fnet2d, the two fcmae_vscyto2d arms and celldiff_2d. pix2pix2d_unetvit is the same GAN with both halves run Z-preserving -- generator input_spatial_size [1,512,512] and patch_size [1,4,4], discriminator preserve_z: true -- not a distinct architecture, matching how celldiff_2d relates to celldiff. M1 had excluded pix2pix for "fixed 3D input size / 3D patch-embed". Half of that is now stale: the generator shares ViTBottleneck3D with CellDiff, so 0126fe6 already gave it per-axis patch extents, and it builds and round-trips [1,512,512] at 65.3M params (3D: 78.1M) with no new code. The discriminator was the real blocker and needed c2d0481. ⚠ Mirrors Run D, NOT the LSGAN baseline that the 3D train.yml leaves compose. The published pix2pix3d weights came from the 12 train_4gpu_modernized.yml leaves: their checkpoints carry loss_type='nonsat', lr 2e-4/2e-4, r1_gamma=10, ema_kimg=10, lecam_gamma=0.3, lambda_l1=10, 201 generator_ema.* tensors and max_epochs 40. Mirroring the LSGAN overlay would have pitted a 2D LSGAN arm against a 3D Run-D arm and confounded geometry with the recipe. The three Run-D deltas are baked into the overlay rather than repeated in every leaf, and checkpoints monitor loss/validate_ema as in 3D. Twelve leaves: 4 organelles x {ipsc_confocal, a549_mantis, joint}. All read the offline 5-plane `_focus.zarr` slab stores at z_window_size=1, so the manifest resolver is disabled and the data fields are inline. The target fragment's [13,624,624] weighted crop is list-replaced to [1,624,624]; NormalizeSampled(fov_statistics) is inherited unchanged, including in the joint leaves' inline children, where celldiff diverges to MinMaxSampled. Z-flattening follows the fcmae_vscyto2d convention: rotate_range -> [0,0,0] (rotation about axis 0 mixes Y and Z, degenerate at Z=1), scale_range Z -> [1,1], sigma_z -> [0,0], crops -> [1,512,512]. XY augmentation policy is untouched so 2D-vs-3D isolates geometry. Global batch is held at the 3D arm's 16 samples/step: 4-GPU DDP x batch_size 4 single-set, and batch_size 2 x num_samples 2 per joint child since BatchedConcatDataModule does not divide. A GAN cannot absorb a batch change quietly -- DynacellGAN derives the G-EMA decay per-step from the global batch size -- and collapsing onto one rank was measured and rejected: on an A40 (44.7 GiB) at [1,512,512] with bf16 + two AdamW states, peak is 13.3 GiB at bs=4 and 25.0 GiB at bs=8, and bs=16 OOMs. That 13.3 GiB/rank is also why the leaves narrow the hardware_4gpu constraint OFF h100|h200, the opposite of the 3D leaf's h200 pin, and cut mem 512G -> 128G for the 5-plane stores. Both go in the leaf body, since launcher_profiles compose after model overlays. The predict overlay carries both documented pix2pix predict traps: predict_overlap Z must be 0 because window_starts requires overlap < patch, and ema_kimg must stay unstated so engine.py rebuilds the EMA shadow from the checkpoint's own keys instead of silently dropping 201 tensors. sliding_window is still needed at Z=1 -- the reason is lateral, the ViT is fixed at 512x512 and A549 is 640x960. All 12 leaves resolve through submit_benchmark_job.py and were checked on the resolved config, not just the exit code: Z=1 geometry, every Run-D knob, 4x4=16 global batch, focus-store data paths, flattened crops and augs, the ema checkpoint monitor, and the rendered sbatch directives. generate_grouped_eval_configs_test.py 44 passed, 1 failure pre-existing (the 6 HEK leaves' cross_condition_probe key). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…y wall All four TIMEOUTed at hardware_4gpu.yml's 4-day cap and were never resumed, leaving them at 102/200, 86/200, 114/200 and 118/200 epochs: jobs 31822521 (nucl scratch), 33631899 (nucl pretrained, third attempt after 31822355 CANCELLED and 31822498/33557666 FAILED), 31822536 (memb scratch) and 31822529 (memb pretrained) — every one TIMEOUT with elapsed 4-00:00:11 to 4-00:00:29 against a 4-00:00:00 limit. This corrects the campaign's own record, which had these four as "no wall hit — cause unestablished" and gated their resubmission on diagnosing an unexplained stop. There is nothing to diagnose: they are the same failure as the ER/mito four, which already carry this profile. Measured 1.22-1.40 epochs/h, i.e. 143-164 h to reach max_epochs=200. That makes the 7-day wall only 1.02x the slowest requirement, so the profile header now says explicitly that these four fit inside it only because they are resumed (80/81/69/66 h from their current epochs) and that a from-scratch nucleus/membrane joint fit needs a longer wall than this profile provides. _LONG_WALL_TRAIN_LEAVES grows from four leaves to eight with the measured rate attached, as test_only_measured_slow_fits_get_the_long_wall requires; the test guards both directions, so this is the only way the list can move. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…d peak The 12 pix2pix2d_unetvit fit leaves set constraint="a40|a6000|l40s|h100|h200", which test_4gpu_train_leaves_inherit_a100_exclude rejected outright — the leaves shipped in de9ad32 without this suite being run, so it has been red since. The constraint is deliberate and the guardrail's stated reason does not cover it: A100 is excluded for repeat NCCL coordination hangs, and this widening still excludes A100. The >=80 GB floor is a proxy for "cannot OOM", and pix2pix2d peaks at 13.3 GiB/rank at batch_size 4 (measured on the interactive A40, 44.7 GiB usable) — 3.4x headroom on the smallest card admitted. All three added cards can satisfy --nodes=1 --gpus=4 in the gpu partition: gpu-c-1 has 8 a40, gpu-b-[1-6] 4 a6000 each, gpu-g-2 4 l40s (gpu-g-1 has 3 and is simply never selected). The payoff is backfill — each fit is ~35 min (job 35148927 COMPLETED 0:0 in 34:38 at 40/40 epochs), so restricting 12 of them to Hopper queues them behind the 4-day and 7-day fits competing for the same cards. Rather than loosen the rule for everyone, the exception is an explicit allowlist carrying that measurement, mirroring _LONG_WALL_TRAIN_LEAVES. Every other 4-GPU leaf stays pinned to h100|h200, A100 stays excluded for the listed leaves too (no measurement can license a NCCL hang), and a widening that traded Hopper away instead of adding to it now fails. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…finished fits First 8 of the 48 pix2pix2d predict leaves: iPSC-trained and A549-trained nucleus, each against 3 A549 conditions + iPSC. The remaining 10 arms are still fitting and get theirs as each one lands, so the unblocked majority predicts alongside the tail instead of behind it. An arm contributes leaves only once its run log carries `Trainer.fit` stopped: — never a checkpoint filename, which names a best-by-monitor epoch and on the fnet2d fits lagged the true final epoch by 73 (project_checkpoint_write_halt). Both arms here logged max_epochs=40 reached. The baked ckpt is the monitored callback's own best_model_path out of last.ckpt, so it is the epoch the recipe selected on loss/validate_ema: epoch=39-step=20000-loss=0.724 (iPSC) and epoch=39-step=12000-loss=2.340 (A549). Source normalization is NormalizeSampled(Phase3D, fov_statistics, mean/std), reproducing what the fit inherits from targets/nucleus.yml — a MinMax source here (the celldiff_2d idiom) would silently degrade every prediction. dataset_ref overrides target to h2b on the A549 sets only; the iPSC manifest has no h2b and raises TargetNotFoundError if it leaks in. No z_reduction: at z_window_size=1 each plane is its own window, so there is nothing to reduce across, unlike the 3D leaves' 8-plane windows. Composition-verified: all 8 resolve to the intended test store and their own arm's checkpoint at z_window_size=1 with predict_method=sliding_window. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Third arm to finish (`Trainer.fit` stopped: max_epochs=40 reached), so its 4 leaves go out now rather than waiting on the remaining 9 fits. Baked checkpoint is the run's best loss/validate_ema epoch, epoch=39-step=12000-loss=0.648, read from the monitored callback's best_model_path inside last.ckpt. Composition-verified: all 4 resolve target_channel=Membrane against the combined dual_nucl_memb test stores (and iPSC cell.zarr) at z_window_size=1. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Fourth arm to finish. Emitted and submitted by the auto-submitter rather than by hand: it re-runs the leaf generator every 5 min and fires an arm's 4 predicts as soon as that arm's run log carries `Trainer.fit` stopped:, so no arm waits on the slowest of the 12 fits. Baked checkpoint is the run's best loss/validate_ema epoch from the monitored callback's best_model_path inside last.ckpt. Jobs 35150143, 35150144, 35150146, 35150147. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…eaves Fifth and sixth arms to finish, both emitted and submitted by the auto-submitter in the same pass, so they land as one commit rather than two identical ones. Jobs 35150155-158 (joint nucleus) and 35150171-175 (A549-trained ER). The joint arm is the first pix2pix2d leaf set whose fit pooled iPSC + A549, and it needs no batch-semantics adjustment at predict time: the joint/single-set batch_size asymmetry is a BatchedConcatDataModule train_dataloader property, and predict runs a plain HCSDataModule at batch_size 1 from the predict overlay. ER uses targets/er_sec61b.yml and carries no dataset_ref target override — unlike nucleus/membrane, A549 does not re-key ER by gene, so adding one would raise TargetNotFoundError against the iPSC manifest. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Sixth of the twelve pix2pix2d arms to finish its fit. The mito/a549_mantis run reached `Trainer.fit` stopped: `max_epochs=40` reached, so its four predict leaves are now generatable: the three A549 conditions plus the iPSC out-of-domain transfer. Completion is read from the run log, never from a checkpoint filename -- Lightning writes the stop line to stderr, so slurm/*.err and wandb/run-*/files/output.log are searched alongside slurm/*.out. The baked checkpoint is the run's best-by-monitor epoch (epoch=35-step=7848-loss=1.245), not last.ckpt: save_top_k means the highest retained epoch tracks the monitor, not progress. The iPSC leaf carries no dataset_ref override. Only the A549 test sets need one, and leaking the h2b/caax target into an iPSC leaf raises TargetNotFoundError: target not found in dataset 'aics-hipsc'. No z_reduction, unlike the 3D pix2pix leaves: at z_window_size=1 each plane comes from exactly one window, so there is nothing to reduce. predict_method=sliding_window stays, because the ViT generator is fixed at 512x512 by its positional embeddings while A549 is 640x960. test_benchmark_config_composition.py: 1028 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…rse them 5a25142 registered ipsc__bf/a549__bf in _CANONICAL_TRAIN_SET_TO_BUCKET so the brightfield-input ablation would get its own buckets instead of colliding with the phase predictions. But main() and emit_readme() iterate _ORGANELLES x _TRAIN_SETS, and that tuple still listed only the three phase labels -- so the 32 bf conditions parsed cleanly, landed in `buckets`, and were then never written. No error, not even the "skipping empty bucket" line: just four silently absent leaves. Adding both labels to _TRAIN_SETS makes them emittable. A dry-run against the live tree now reports er/nucleus x {ipsc,a549}_bf_trained at 8 conditions each (2 models x 4 test sets = the full 32), and mitochondria/membrane at 0, correctly skipped -- the ablation only covers nucleus and ER. No collision with the phase arms, verified rather than assumed: save_dir_for and pred_cache_dir_for both key on train_set_canonical, so bf lands under ipsc__bf/a549__bf while phase stays at ipsc/a549; condition_name keys on the bucket label, so the two are distinct there too. benchmark_dataset_ref and _gt_cache_dir_for key only on test_set/organelle, which is correct -- __bf swaps the model *input* channel while the fluorescence GT is the same store. Also drops the now-false docstring line on parse_zarr_name still naming a549__bf as an out-of-scope token; the ablation dirs are the live example. test_every_registered_bucket_label_is_emittable pins the invariant that was missing: every label in the bucket table must appear in _TRAIN_SETS. Asserted as a subset relation, so a future half-registered bucket fails rather than disappearing. Suite: 42 passed, 4 deselected. This is capability only -- no leaves emitted here. 12 of the 32 bf predicts are still running, and the generator's only freshness gate is pred_path.is_dir(), which passes on a half-written store, so emission and eval wait on a chunk-count check. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Eighth of the twelve pix2pix2d arms to finish its fit: mito/ipsc_confocal reached `Trainer.fit` stopped: `max_epochs=40` reached, so its in-domain iPSC predict plus the three A549 transfer conditions are generatable. Baked checkpoint is the run's best-by-monitor epoch (epoch=35-step=18000-loss=1.045), identical across all four leaves -- one fit, four test sets. No dataset_ref override in any of the four, which is correct rather than an omission: only nucleus and membrane need one, because their A549 target token (h2b/caax) differs from the iPSC one. TOMM20 is spelled the same in both manifests, so the target resolves without help. Leaking an override into an iPSC leaf is what raised TargetNotFoundError earlier in this arm's generation. test_benchmark_config_composition.py: 1032 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Ninth of the twelve pix2pix2d arms to finish its fit. er/joint reached `Trainer.fit` stopped: `max_epochs=40` reached, so its in-domain iPSC predict plus the three A549 conditions are generatable. Baked checkpoint is that run's best-by-monitor epoch (epoch=34-step=22960-loss=0.671), from the joint model dir, and all four leaves share it. Its step count is the joint tell: 22960 steps over 35 epochs against the iPSC-trained arm's 18000 over 36, because BatchedConcatDataModule pools iPSC + A549 and so runs more steps per epoch. Prediction stores land under er/pix2pix2d_unetvit/joint/<test>/, the canonical joint path -- not the CellDiff-R2 joint_predictions/ layout, which is that family's own exception. test_benchmark_config_composition.py: 1036 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Tenth of the twelve pix2pix2d arms to finish its fit. mito/joint reached `Trainer.fit` stopped: `max_epochs=40` reached, so its in-domain iPSC predict plus the three A549 conditions are generatable. Baked checkpoint is that run's best-by-monitor epoch (epoch=39-step=29280-loss=1.072), shared by all four leaves. Unlike the other joint arms its best epoch is the last one (39/40), so save_top_k retained the final epoch here -- still read from the monitored callback rather than assumed from the filename. Prediction stores land under mito/pix2pix2d_unetvit/joint/<test>/, the canonical joint path. test_benchmark_config_composition.py: 1040 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2cffa41 fixed the brightfield buckets going unemitted by growing _TRAIN_SETS to match _CANONICAL_TRAIN_SET_TO_BUCKET, then added a test to assert the two agree. That treated a duplication as something to guard rather than something to remove: the same set of bucket labels was written out twice, and only a human (or the new test) kept them in sync. Derive it instead -- tuple(dict.fromkeys(table.values())) -- so a label the table registers cannot be omitted from emission at all. dict.fromkeys dedupes the many-to-one labels (a549/a549__deconv both map to a549_trained) while preserving first-occurrence order. Verified behavior-neutral: the derived tuple is ('ipsc_trained', 'joint', 'a549_trained', 'ipsc_bf_trained', 'a549_bf_trained') -- byte-identical to the hand-written one, so the generated README and bucket emission order do not move. Achieving that needed the two joint rows moved above the a549 rows in the table literal, which is safe for every consumer: the table is read only via .get(), .values() and sorted(), never iterated for meaning. Since literal order is now the only source of README row order, that is called out in a comment above the table. _TRAIN_SETS moves below the table because it now depends on it. test_every_registered_bucket_label_is_emittable is kept rather than deleted: the subset relation is now true by construction, so the test's job becomes failing loudly if anyone re-hardcodes the tuple, which is what would reintroduce the bug class. It also now pins the order, so a table reorder is a deliberate choice rather than a surprise README diff. generate_grouped_eval_configs_test.py 42 passed; full config suite 1082 passed, 119 skipped, 0 failed. --dry-run against the live tree emits the same 5 buckets per organelle in the same order. Found by a reuse/simplification/efficiency/altitude review pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…w rationale All 40 pix2pix2d predict leaves restated, in a 3-line header paragraph, why the overlay sets predict_method=sliding_window and predict_overlap Z=0. The overlay already documents it -- and documents it better, because it carries the caveat the copies drop: "(the iPSC test is a single 512x512 tile)". Without that caveat the restatement is wrong in the 10 predict__ipsc_confocal.yml leaves, where it asserts the reason is that "the A549 test set is 640x960" for a leaf that never touches A549. The real reason sliding_window is still needed there is overlay-level, not leaf-level, which is exactly why the overlay owns it. Replaced with a pointer at model_overlays/pix2pix2d_unetvit_predict.yml. This also matches the sibling ViT family: the pix2pix3d_unetvit predict leaves never restate this overlay-owned fact either. Comment-only -- verified no changed line lacks a leading '#', so every composed value is untouched. The scratch generator that emits these leaves was fixed in the same pass, so re-running it now reproduces all 40 files byte-identically (md5 of the set unchanged across a regeneration) instead of reverting them on the next poll. Full config suite: 1082 passed, 119 skipped, 0 failed. Found by a reuse/simplification/efficiency/altitude review pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

… NaN the gradient Three joint FCMAE resumes died with OptimizerHealthError 109-169 steps after restoring a healthy checkpoint: finite forward loss, every one of the model's 32.1 M parameter gradients NaN, zero inf, recurring roughly 1 step in 6-27. Resuming from earlier checkpoints does not help and is not even monotonic in checkpoint age (119 -> 169 steps going back 6 epochs on one arm, 117 -> 109 going back 3 on another), which rules out a poisoned weight state; the source checkpoints are clean (0 non-finite across 252 tensors and the optimizer state). detect_anomaly on the real run named MulBackward0 for the sigma_x * sigma_y product feeding the Cauchy-Schwarz bound. That bound was added 2026-07-23 in a556bc9 and did not exist when these checkpoints trained, which is why the same data and config diverge now. The trigger is precision, not structure. On the captured failing batch the input gradient is 0.02% NaN under 16-mixed and exactly 0.00% in fp32: the floored SSIM denominator produces gradients around 1e8, which overflow fp16 to inf, and a later multiply turns those into NaN. _compute_ssim_and_cs_bf16 already selects its own precision on purpose -- bf16 convolutions, fp32 for everything after -- so an enclosing autocast re-entering it defeats that choice. Disable autocast around the call. Two alternatives were measured and rejected: - detaching the bound clears the NaN but drops gradient agreement with the fp32 monai reference from cosine 0.99 to 0.79, because the clamp binds often enough on ordinary inputs that the bound carries real gradient. - splitting into _safe_sqrt(sigma_x) * _safe_sqrt(sigma_y) leaves the NaN entirely, confirming the large product is not the mechanism. The forward moves by ~3e-6 on the captured batch (it now always takes the fp32 path). ms_ssim_25d is not read by the dynacell evaluation pipeline, so no reported metric changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Completes the 12-arm pix2pix2d family at 48/48 predict leaves. This arm was the last fit to finish (35151606, COMPLETED 0:0 in 6:53:57), so its four leaves are the ones the auto-submitter emitted last. The fit is verified complete on the Lightning stop reason rather than sacct alone: `Trainer.fit` stopped: `max_epochs=40` reached, epoch 39 / step 32000, and the baked checkpoint carries all 201 generator_ema.* tensors with 0 of 456 non-finite. The EMA tensor count matters here -- a recipe without ema_kimg silently drops them and the predict then runs the raw generator. It took 6:53:57 against 1:13-1:23 for the other joint arms only because it landed on gpu-c-1, an A40 without NVLINK, while the others got h100/h200 nodes. That spread is expected: these leaves deliberately narrow off h100|h200 to backfill better, since the 2D GAN needs just 13.3 GiB/rank. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…uckets Takes each bucket from 16 to 20 conditions (48 new conditions), completing the in-focus 2D track at 5 families. pix2pix2d was held out until now for the documented reason: the generator's only freshness gate is pred_path.is_dir(), which passes on a partially-written store. Store completeness gate (.tmp/pix2pix2d/verify_predict_stores.py), a stronger one than celldiff_2d got. Per array, expected chunk count = prod(ceil(shape / chunks)) read analytically from that array's own zarr.json, compared against the files actually present -- this catches a truncated write inside a position without needing a reference family, which a fnet2d-relative count could not do, since it cannot detect a shortfall shared with its own reference. Paired with a position-set comparison against a family already known complete, because a store missing whole FOVs is internally self-consistent and its own metadata can never reveal the gap. All 48 stores passed both: 100 positions per iPSC bucket, 12 per A549 bucket, every array fully chunked. sacct: all 48 predicts COMPLETED 0:0. Condition blocks are lifted verbatim from a generate_grouped_eval_configs.py --out-root <scratch> run, never hand-written. Each goes immediately after its test set's fnet2d block, which is where the generator itself puts it: its within-test-set order is celldiff_2d, celldiff_r2, vscyto2d, unext2_2d, vscyto3d, unext2, fnet2d, fnet3d, pix2pix2d, pix2pix3d, so among the 2D-only models pix2pix2d sorts last. An integrity gate asserted per file that deleting the 4 inserted blocks restores the original bytes exactly. That is what proves the 16 already-evaluated conditions were untouched -- they cache-skip only if their text is unchanged. Header and condition edits are gated separately so a header rewrite cannot stand in for a condition change. Also verified: 240 conditions across the 12 buckets have 240 unique pred_cache_dir and 240 unique save_dir, all 240 pred_paths exist on disk, and each bucket has 20 unique condition names. force_recompute.final_metrics stays false, so the 16 pre-existing conditions cache-skip -- fnet2d, both FCMAE-2D arms and celldiff_2d all have feature_metrics.csv in 48/48 of their save dirs (checked, not assumed; pix2pix2d is 0/48). Only the 4 pix2pix2d conditions pay the cold deep-feature extraction, which needs --constraint a100_80|h100_80|h200. Headers updated in the same files: scope 16 -> 20, the gate actually applied, and the corrected cache note. That is one concern with the fold. Tests: generate_grouped_eval_configs_test.py + test_benchmark_config_composition.py + test_paths.py = 1898 passed, 119 skipped, 0 failed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

``_final_metrics_cache_valid`` has required the SI_* columns since 36cbfdf ("reject a pixel cache that carries only one metric scaling"), but this fixture's pixel row stayed ``{"metric": "pixel"}``. The gate therefore invalidated the cache it was built to prove reusable, and ``test_evaluate_model_reuses_cache_without_feature_metrics`` has been failing since 2026-08-03. It went unnoticed because test_evaluation_pipeline.py is one of the files the default ``pytest`` invocation does not collect. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Metric values depend on which cubic is installed and nothing recorded it. Two eval venvs coexisted from 2026-06-30 (cubic 0.8.0a2) and 2026-07-29 (0.9.0a1), and because run_eval_direct.slurm defaulted to the older one, jobs run after the switch silently kept using it. Of the 121 surviving eval logs, 62 ran 0.8.0a2 and 59 ran 0.9.0a1 — the corpus on disk is interleaved, and mtime cannot separate it. Measured across the pin on one real A549 nucleus pair (2 FOVs, 110 values), four columns move: Z_FSC_Resolution +30.9%, XY_FSC_Resolution +16.7%, Spectral_PCC <=0.57% (sign varies), FRC_Resolution +0.52%. Everything else is bit-identical: PCC, SSIM, NRMSE, PSNR, every SI_*, PerCell_*, every AP_*/mAP/instance_dice, MicroMS3IM (5e-8) and the CP feature columns (2e-15). The 0.9.0a1 bug that moved PSNR +45 -> +59 in its own fixture — normalize combining with the scale-invariant path — does not reach us, because dynacell never passes both in one call. So: - provenance.py declares REQUIRED_CUBIC_VERSION and check_cubic_pin() fails a run whose environment disagrees, instead of writing plausible values from a different numeric stack. - save_metrics writes metrics_provenance.json beside the CSVs and _final_metrics_cache_valid refuses an unstamped or foreign-stamped cache. Unstamped means "written before the stamp existed", i.e. an unknown pin, so it must recompute — deliberately unlike the per-extractor preprocess_version bootstrap, which treats an untagged entry as unconstrained. provenance_test.py pins the constant to every pyproject pin so the declared version and the installed one cannot drift apart silently. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

DYNACELL_EVAL_VENV fell back to the cpdino-eval venv (cubic 0.8.0a2), which stayed the default for a month after the 0.9.0a1 venv existed. Any job that did not export the variable therefore produced 0.8.0a2 FSC/FRC/Spectral_PCC values with nothing on disk to say so — 62 of the 121 surviving eval logs. check_cubic_pin() now fails such a run loudly, but the default still has to point at the version pyproject declares. Also drops the stale version numbers from the surrounding comment; the ImportError it documents is a property of the workspace .venv, not of any particular cubic release. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

There is no instance_ap*.csv. Globbing for one returns zero hits for every model family, including families evaluated weeks ago, which reads as "AP was never computed" rather than "wrong filename" — a wrong conclusion about the corpus rather than a visible failure. Asserts the AP_*/mAP/instance_dice columns land in the mask CSV via the real save_metrics, and that no instance_ap*.csv is produced. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

mito/pix2pix3d_unetvit/ipsc/ holds four prediction stores with metrics on disk, but mitochondria_ipsc_trained listed only five models, so no grouped bucket ever reached them -- pix2pix3d appeared in mitochondria_a549_trained, mitochondria_joint and mitochondria_hek_a549xy_pix2pix but not here. The gap surfaced during the cubic 0.9.0a1 re-pin sweep: 550 of 564 canonical eval dirs now carry a metrics_provenance.json stamp, and these four are among the 14 that do not, because a sweep can only visit conditions a bucket declares. That is a bucket defect, not a rerun away. Takes the bucket 22 -> 26 conditions. The four blocks are copied verbatim from generate_grouped_eval_configs.py run into a scratch --out-root, never --out-root at the repo: the generator emits 46 conditions here because it folds the 2D track in, and this leaf is deliberately the 3D-only slice. Verified before writing that all 22 pre-existing blocks are byte-identical to the generator's, so the insert cannot silently rewrite a neighbour, and that the resulting condition order equals the generator's own 3D-only ordering (pix2pix3d between fnet3d and unetvit3d in each test set). Composed through the real Hydra loader: 26 conditions, and for each new one pred_path, gt_path, cell_segmentation_path, gt_cache_dir and pred_cache_dir all exist, with per-test-set spacing resolved correctly (A549 0.174/0.1494, iPSC 0.29/0.108) and the iPSC arm reading TOMM20.zarr rather than cell.zarr. generate_grouped_eval_configs_test.py and generate_instance_ap_eval_configs_test.py pass (62 tests). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Speedup pilot P2: the nucleus FNet-3D vscyto3daug v2 baseline branched at 38 / 60 / 100% of its 200k-step constant-LR budget and cooled linearly to zero LR, each predicted from its final weights. Registers the three PAPER_KEY tokens so they do not collapse onto v2, and adds the ctrl6 wave (cool38 and cool60 all legs, cool100 three legs; its a549__denv store is still predicting). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

Speedup pilot P3: the UNeXt2-3D membrane segaux_seed1 arm forked from the v2_seed1 baseline's epoch-123 checkpoint instead of trained from scratch, predicted from its best-val epoch 149 (inside the e123-e178 range of the runs it is compared with). Registers its PAPER_KEY token and adds the ctrl7 wave, all four legs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

…into dynacell-spotlight-v2-eval-waves

wave3n: the Phase B2 FNet-3D vscyto3daug nucleus loss-weight sweep (segaux at 0.5x and 2x seg_aux_weight), both at --ckpt best epoch 51, inside the baselines' epoch 64 / 74-control range; all four legs. ctrl6b: the last P2 budget-control leg, cool100 a549__denv. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

pix2pix3d_unetvit_segaux_sauna_seed1 re-draws the membrane SAUNA arm with seed_everything: 1 (SEED_SOURCES), at the same calibrated seg_aux_weight 2.9. Draw 1 passed the B1 rules with an A549 FG_PCC gain over both baselines and both segaux draws, which rule R3 does not decide on a single draw. Registers the PAPER_KEY and grouped-eval tokens and generates the train leaf and four predict leaves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

…ed flags (#529) trainer.benchmark: true on the FCMAE (UNeXt2-3D) fit overlays and trainer.benchmark + model.init_args.channels_last on the FNet3D fit overlays. Measured on H200 with the real dataloader: UNeXt2-3D 4-GPU DDP 0.930 s/step vs 4.322 1-GPU default (compile slower under DDP, so off); FNet-3D 0.493 vs 0.918 s/step; training-loss numerics within the replicate floor. Predict overlays are untouched, so predict leaves resolve unchanged. A data-driven test keyed on the composed architecture pins the flags on every 3-D FCMAE/FNet3D train leaf, keeps them off every other train leaf and off these families' predict leaves. Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

The four fit overlays carried two copies of the same 4-5 line rationale (found by /simplify on #529). Keep it in the primary overlay of each family, cite the 1-GPU measurement that isolates the flag (0.900 vs 4.322 s/step; the 4-GPU row mixed in the GPU count), and point the copy overlays at it. The fit-only reason is the user's decision to keep predicts on the defaults their stores were made with; "kernel choice only" argued the opposite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

From /simplify on #529: list only the flags each family sets (no None placeholders or default dict), have the helper return the expected flags directly, and name the family FCMAE (a separate timm UNeXt2 architecture is not covered). Drop the predict-leaf test: no predict leaf composes a *_fit.yml overlay (0 of 1219), so it could not fail, and it skipped 722 of its 1219 cases. Mutation check: benchmark false in fcmae_vscyto3d_fit.yml fails 35 train leaves; composition file 2141 passed, 190 skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

/code-review on #529: the FCMAE overlay said the cudnn.benchmark train loss was within the replicate floor, but the measured gap is 1.25x the floor, from one replicate pair over 400 steps inside the LR warm-up (speedup RECOMMENDATIONS.md s7). Only FNet3D (0.92-0.95 over 3,000 steps) supports "within the floor". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

… focus2d eval (#526) * feat(viscy-data): predict only listed Z planes per position and timepoint A 2D model predicts every plane of the full-Z test store, while the 2D benchmark scores only the focus slab, so most of its predict time and storage goes to planes nothing reads. HCSDataModule.predict_z_planes takes a JSON plane file (position -> per-timepoint plane list) and restricts the predict windows to it (z_window_size=1). The writer allocates the full source extent so the store stays aligned plane by plane with its source (skipped planes keep fill value 0), records the predicted planes per channel (viscy_predicted_z_planes, cleared again by a full run) so a reader can refuse to score zeros, and carries z_planes_sha256_12 in the completion marker so a resume never mixes two plane selections. The launcher's --resume-predict hashes the same file, read as JSON to keep the launcher free of torch. Measured on fnet2d ER, A549-trained, A549 mock (A40): the slab3 store equals the existing full-Z store on all 360 slab planes (max |diff| 0) and holds no other plane; 0.78 vs 12.2 GB on disk; predict loop 60 s vs 930 s at the same ~6 windows/s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * feat(dynacell): score the 2D benchmark on one focus plane (focus2d) The 2D benchmark scored only ER/mito Dice on the focus plane (segmentation.semantic_focus_halfwidth); every other metric still integrated the 2D model's per-plane predictions over the full volume, and a 2D model that predicts only its focus slab could not be scored at all. focus2d.halfwidth replaces that key and moves every metric to one GT-anchored plane per FOV/timepoint (segmentation.focus_anchor): - pixel metrics, MicroMS3IM (and its leaf calibration pool) and CP on the plane itself; 2D SSIM / Spectral PCC, FRC replaces the XY/Z FSC - deep features and per-cell similarity on the 2h+1 slab around it - semantic masks segmented on the slab and scored on the plane; instance targets must use the same slab (validated) GT/pred caches move to <cache_dir>/focus2d_h<h>/ and record the plane recipe, refusing another: the CP cache key carries no z geometry, so a shared dir would silently score 2D prediction rows against cached 3D GT rows. CP uses its own <target>__focus2d_h<h> reference, built by build_cp_reference.py --focus2d from the focus2d leaves' per-dataset plane recipes. The recipe is stamped in metrics_provenance.json and compared both ways, and a grouped bucket cannot mix modes. precompute-gt build.focus_planes writes the plane file a 2D model predicts with, resolved by the same function the eval uses. The eval refuses a plane-restricted store that lacks its slab (e.g. another anchor) and any plane-restricted store under full-volume scoring. The eight ER/mito slab3 buckets move to focus2d.halfwidth=1; their masks are unchanged, and their pixel rows are now on the plane. Measured on fnet2d ER, A549-trained, A549 mock: focus2d eval of the full-Z store and of its slab3-only re-predict agree on all 120 pixel and mask rows (max |diff| 0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * feat(dynacell): predict 2D leaves on the focus2d slab only Every 2D predict leaf (the five benchmark families and their ablation arms) now sets data.init_args.predict_z_planes to focus2d_planes/<organelle>/<test>__h1.json and writes prediction_focus2d_h1.zarr beside the existing full-Z prediction.zarr, which stays untouched. A 2D model then predicts only the 3-plane slab the focus2d eval reads: ~1/16 of the planes on A549 (48 -> 3), ~1/13 on iPSC. Setting predict_z_planes to null (and the store back to prediction.zarr) predicts every plane as before. The 150 Spotlight-v2 arm leaves are regenerated from their edited baselines by tools/generate_spotlight_v2_leaves.py (--check: 407 leaves, 0 stale), so the ccond arm reads its FNet segaux mask from that arm's slab3 store on the same planes. The plane files are written by precompute-gt build.focus_planes; the predict and eval stores they index are the same file for all 16 organelle/test pairs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * feat(dynacell): add focus2d grouped eval buckets for the 2D benchmark grouped/<organelle>_<pool>__2d_focus2d holds the 20 conditions of each canonical <organelle>_<pool>__2d bucket (5 2D families x 4 test sets), scored with focus2d.halfwidth=1 and written under eval_focus2d_h1/; <organelle>_phase_floor__2d_focus2d scores the raw Phase3D input on the same planes. Anchors: nucleus_area for nucleus/membrane and iPSC ER/mito, phase_midband for A549 ER/mito (no nuclei channel). They read the existing full-Z prediction stores, which hold the slab plane for plane (a slab3 re-predict scored identically), so no re-predict is needed. Generated by experiments/2026-10-07_2d-vs-3d-cost/make_focus2d_buckets.py (gitignored). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): let focus_plane_stack pass the 3D benchmark through focus_plane_stack(vol, None) now returns vol, so the four "full volume or focus-plane stack" ternaries and the duplicated fov_cp_features call in precompute-gt become one unconditional call each. The plane cut reuses slab_mip with halfwidth 0, the module's documented single-plane slicer, instead of a third slicing idiom. The focus helpers move to the module-level import in pipeline.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): read the eval's GT nuclei through gt_nucleus_volume _process_one_fov kept its own copy of "which store and channel hold the GT nuclei" next to the gt_nucleus_volume helper the calibration and precompute-gt use, plus two inline "the nucleus target is its own nucleus volume" expressions. _gt_nuclei now memoizes the helper, so the eval, its MicroMS3IM calibration and the predict plane file place the plane from one definition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): build the uncovered-slab map in one comprehension The coverage check built the per-timepoint differences and then filtered the empty ones in a second dict; one filtered comprehension does both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): resolve the CP reference halfwidth via read_focus2d_config resolve_cp_reference_path read focus2d.halfwidth with a raw select, a second unvalidated reading of the key: a negative halfwidth named a '__focus2d_h-1' reference instead of raising like the rest of the eval. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): carry focus estimator params into the focus2d CP reference build read_focus2d_config folds the focus.* estimator params into the phase_midband plane recipe, but build_cp_reference only merged three segmentation keys from the leaves, so a leaf overriding focus.* would build a config whose plane recipe differs from the eval's cache. No focus2d leaf sets focus: today, so the built references are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * test(dynacell): fold the hand-built plane-store check into the predict test Both tests asserted that a plane-restricted store scores like the full one; the real HCSDataModule predict already produces such a store, so the two refusal checks move onto it and the zero-filled copy goes, saving two end-to-end evaluate_predictions runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(viscy-utils): canonicalize predicted planes in one helper z_planes_sha256_12 and record_z_planes each spelled out the same sorted, de-duplicated int conversion; _canonical_planes now serves both, so the run hash and the recorded planes cannot drift apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(viscy-utils): derive the writer's position name once write_on_batch_end resolved the same position twice on consecutive lines by two expressions, one through a single-use _position_name helper. One name now indexes both the plate and the plane map, and the skipped-window mask indexes planes directly (fancy indexing needs no sort or de-dup). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): hash the predict plane file through read_z_planes The launcher parsed the plane file's positions itself, a second reader of the schema, on the stated grounds of keeping torch out. Importing the launcher already loads viscy_data.sliding_window (measured: the extra import costs 0.0 s), so it now calls read_z_planes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(viscy-utils): refuse a plane-restricted predict into another run's store overwrite rewrites chunks in place, so a slab predict into a store a full-Z (or other) run wrote left that run's planes around the slab: the store kept its old size and voxels from other weights, and an existing short array was never grown to the source extent. Run positions now must hold no prediction channel another run wrote; rerunning the same plane run (resume) still passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): keep prediction.zarr as the 2D predict leaves' store The slab3 leaves wrote prediction_focus2d_h1.zarr, a name the store grammar (key_from_prediction_store), both eval-config generators and the focus2d buckets never read, so a re-predict would have been invisible to every eval. The store name stays prediction.zarr: the writer records the planes, the eval refuses a full-volume score of a slab store, and the writer refuses a slab predict over a full-Z one. The leaf comment no longer says null switches the store name. The 150 Spotlight-v2 leaves are regenerated (--check: 407 leaves, 0 stale). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): refuse build.focus_planes under limit_positions The plane file holds only the positions the loop visits, so a smoke run with limit_positions atomically replaced the full file and every predict on it then failed on the missing positions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): refuse a full-volume eval of a slab store before calibration The refusal ran per FOV, after the leaf-level MicroMS3IM calibration had already fit on the store's zero planes. evaluate_predictions now scans the recorded planes of every prediction position up front. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): keep the plane file's path out of the predict settings hash prediction_settings_sha256_12 hashed predict_z_planes as a setting, so moving or copying an unchanged plane file changed settings_sha256_12 and --resume-predict refused the store, although z_planes_sha256_12 already identifies the plane selection by content. The path is now loading-only, like data_path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): name --focus2d in the 2D CP reference's rebuild hints A focus2d eval loads <target>__focus2d_h<h>.json, but the missing-file, recipe, dataset and GT-cell-mismatch errors all suggested the 3D build command, so following the hint built <target>.json and the eval still failed. _build_command now takes the reference path and adds --focus2d <h> for a 2D-benchmark reference, read back from the name cp_reference_path gives it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * docs(dynacell): trim the slab3 predict leaves' comment to what the leaf decides The two lines on removing a full-Z store repeated the writer's refusal and the dynacell CLAUDE.md note in 268 copies; the leaf keeps only the switch it owns (null predicts every plane). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(viscy-utils): fold the plane-run refusal into the writer's mixed check The plane-restricted refusal was a second list, close and raise for the rule _cannot_share already expresses; a plane run simply extends that check to its own FOVs, so one condition and one message cover both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): resolve focus2d once per evaluate_predictions call Three branches of evaluate_predictions each re-read the focus2d recipe from the config; one local now serves the up-front refusal, the MicroMS3IM calibration and the warm-pass skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * refactor(dynacell): parse the CP reference's focus2d name beside its builder _build_command recovered the halfwidth with a regex that restated the <target>__focus2d_h<h>.json format cp_reference_path defines; the inverse now lives next to it in paths.py, so a rename of the suffix cannot silently drop --focus2d from the rebuild hints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * docs(dynacell): drop a comment that restates the limit_positions error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(viscy-utils): let a slab predict replace another run's same-slab store The plane-run refusal fired on any marker mismatch, so re-predicting a slab store with a new checkpoint (--ckpt best) on the same plane file was refused although it rewrites every plane the store holds. Refuse only when another run's recorded planes differ from this run's (a full-Z store records none), with its own message naming the remedy (remove that store); out-of-run FOVs keep the unchanged mixed check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): refuse a full-volume eval of a slab store before writing The refusal ran inside the with block that opens the eval dir's segmentation_results.zarr with mode="w", so a mistaken full-volume eval truncated that store before refusing. The recorded planes are now checked on their own read of the prediction before anything is opened for writing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(dynacell): refuse an unfinished plane-restricted prediction in the eval A plane run writes its started marker before its first batch but records its planes only on completion, so an interrupted slab predict read as a full-volume store: the focus2d coverage check saw no planes to check and the full-volume refusal saw no plane restriction, and both scored unwritten planes as zeros. plane_predict_unfinished flags a marker that names a plane selection with no planes recorded, and evaluate_predictions refuses such a store up front in both modes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc * fix(viscy-utils): tell an unfinished plane predict by its marker's source_shape plane_predict_unfinished keyed on missing recorded planes, but a re-predict of the same slab replaces only the marker on start and keeps the previous run's planes until it completes, so an interrupted one looked finished and a focus2d eval could score a mix of two runs' planes. A plane run's marker gains source_shape only on completion, so its absence is the signal in every case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ckets Phase B2: the nucleus FNet-3D vscyto3daug segaux_halfw arm lost A549 Dice on all three legs when predicted from its best-val epoch 51. Its re-prediction from latest-epoch=76-step=200000 (the e74 stage-control floor) tests whether that loss is a checkpoint-stage artifact. Registers its PAPER_KEY token and adds the ctrl8 wave, all four legs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

* feat(dynacell): score interior-only foreground metrics by default Spotlight v2 judged models on foreground-limited pixel metrics, because whole-image pixel metrics are dominated by background voxels. Turn the FG_* columns on in eval.yaml with a hard GT interior mask (feather_sigma_um: 0.0); feather_sigma_um: null keeps the per-target feathered recipe from metrics.FOREGROUND_SIGMAS_UM. The weight map is built from the GT alone and only weights the scoring; neither image is zeroed, so whole-image columns are untouched. FOREGROUND_METRICS_VERSION is not bumped: no FG_* value moves for a given recipe, and the recipe is already stamped in metrics_provenance.json, so an existing save dir recomputes its final metrics once on its next run. Hand-built test configs omit the block, so a compose test of the real eval.yaml pins the new default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(dynacell): give nucleoli and lysosomes foreground sigma defaults With FG metrics on by default, an eval whose target has no FOREGROUND_SIGMAS_UM entry raises before scoring, and nucleoli and lysosomes are accepted targets without one. They take mitochondria's sigmas (a region mask around compact or punctate bodies), marked untuned because DynaCell has no GT for them. A composition test now resolves the default FG settings for every accepted target; the null-sigma error test moves to a target outside the table. New entries move no existing FG value, so FOREGROUND_METRICS_VERSION stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * test(dynacell): keep the otsu sigma test on a target with no default test_otsu_source_resolves_no_smoothing_sigma switched to lysosomes to prove the otsu source needs no smooth_sigma_um default. This PR gives lysosomes a default, so the check passed whether or not otsu looks one up. Use golgi, as the sibling resolution test already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * test(dynacell): pin the FG default for every target from the evaluator's list Merge the two new FG-default tests into one parametrized test that checks the exact resolved settings per target. The expected smoothing sigma comes from FOREGROUND_SIGMAS_UM, and the target list from the new segmentation.EVAL_TARGETS, which prepare_segmentation_model now uses too, so a target added to the evaluator is covered automatically. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * refactor(dynacell): derive the nucleoli and lysosomes FG sigmas from mito They take mitochondria's sigmas by design; copying the literal would let them drift silently if mitochondria is retuned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * docs(dynacell): state the FG default once eval.yaml named FOREGROUND_SIGMAS_UM twice in one comment, and the tables.py comment copied the default, which is how its old "off by default" went stale. The default now lives in eval.yaml and the README defaults table only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * fix(dynacell): copy, not alias, the nucleoli and lysosomes FG sigmas The previous derivation bound both entries to mitochondria's dict, so an in-place edit of one (a test monkeypatch or a sensitivity sweep) changed all three. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J * docs(dynacell): describe the default FG weight as the hard interior mask eval.yaml and the README metrics table still called the weight a soft map; under the new default (feather_sigma_um 0) it is binary, feathered only when feather_sigma_um is positive or null. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

The *__2d_focus2d buckets from #526 carry no "Auto-generated by" stamp: they were emitted by an experiment script outside the tracked generators (experiments/2026-10-07_2d-vs-3d-cost/make_focus2d_buckets.py), so a regen of the generated set never rewrites them. #526 branched before the guard landed in 3ae322e and CI has no dynacell job, so the merge left test_every_grouped_leaf_is_generated_or_registered_hand_written failing on dynacell-models, naming exactly these 16. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

…gistry 3ae322e built _HAND_WRITTEN_BUCKETS from the main checkout's directory listing, which holds 26 grouped buckets that have never been committed (*__a549only, *__ipsconly, *_celldiff_r2_*_later, the 2D sample- and voxel-matched finals, er_a549_trained__temporal_rescore, ...). In any clean checkout they do not exist, so test_registered_hand_written_buckets_all_exist_and_are_unstamped failed on dynacell-models naming all 26. The registry records hand-written buckets in the repo. A checkout that still carries those untracked dirs now fails the other test, naming them, which is the guard doing its job: commit the ones worth keeping, or remove them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

…uckets The nucleus FNet-3D vscyto3daug segaux arm (draw 1) gained A549 FG_PCC when predicted from its best-val epoch 48, but ctrl8's halfw arm lost that gain by epoch 76 and draw 2 (epoch 57) never showed it. Its re-prediction from latest-epoch=76-step=200000 tests whether the gain is a checkpoint-stage effect. Adds the ctrl9 wave for the three legs whose predicts finished; a549__zikv follows in ctrl9b. The PAPER_KEY token already exists from the mito control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

The a549__zikv leg of ctrl9 (nucleus fnet3d_vscyto3daug_segaux re-predicted from latest-epoch=76-step=200000), split out because its predict finished after the other three legs' evals had started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

…into dynacell-spotlight-v2-eval-waves

B3b of Spotlight v2. The membrane segaux arm's Dice target is a CLAHE+Otsu mask that fills 40-66% of the plane, so the term rewards thick blobs rather than walls; filter-based thin masks (B3) failed validation. This arm keeps the segaux recipe and swaps only the target: cpdino whole-cell outer outlines on the GT membrane, per plane, inside a wall band around the nucleus-area anchor, canonical mask elsewhere. The mask is a new array in a derived training store that mirrors the iPSC cell.zarr (plate metadata and position order identical, image arrays symlinked), so the fit leaf disables the manifest resolver and declares data_path and both channels inline. The thin mask changes the Dice term's gradient scale, so the weight is calibrated on its own: on the same April checkpoint and val batches, read from the derived store, the canonical mask reproduces 0.8609 and fg_mask_cpout gives 0.5533 [0.397-0.886] -> 0.55. The arm keeps its segaux source's 7-day wall. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J

) * feat(viscy-utils): add a segmenter-consistency loss SegConsistencyLoss matches a frozen segmenter's dense outputs on the prediction and the target: MSE on the two flow channels plus binary KL (BCE with logits minus the target entropy) on the cell probability, so the loss is 0 at pred == target. S(target) runs under no_grad on the same crop every step. A 3-D crop is reduced to the slab MIP around the target's nuclear-area plane, as the eval segments it; an optional Ctrl-U weight down-weights pixels where S disagrees with its flipped view. The segmenter stays out of the module tree, so checkpoints and strict ckpt_path loads are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dynacell): wire the segmenter-consistency term into the UNet and GAN engines DynacellUNet and DynacellGAN take seg_consistency + seg_consistency_weight, added next to seg_aux (training loss / generator objective). loss/validate stays base-only; the term is logged as loss/segcons_* (UNet) and loss/g_segcons_* (GAN) in training and loss/validate[_ema]_segcons in validation. The weight has no default: a term with weight None (the leaf placeholder) is refused at build, as is a weight without the term. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dynacell): add a differentiable Cellpose-SAM wrapper CellposeDense returns cellpose flows and cell-probability logits that are differentiable w.r.t. the input, ported from the G4 gate: normalize99 with no-grad statistics, 256-px tiles, taper blend, loaded by absolute path from cellpose's model dir (no silent fallback or download), cpdino refused. With a gradient, tiles run under activation checkpointing (checkpoint_tiles, default 1): a stored fwd/bwd holds ~5.6 GB per tile. Tests: parity with CellposeModel.eval on identical normalized input, finite-difference gradient, checkpointing parity, and one DynacellUNet step through SegConsistencyLoss (GPU + cpsam only). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dynacell): add P1 segmenter-consistency fine-tune arms (nucleus) Spotlight-v2 generator arms segcons / segcons_ctrl on FNet-3D vscyto3daug, UNeXt2-3D and pix2pix2d nucleus: both fork the baseline recipe's final checkpoint weights-only (model.init_args.ckpt_path) and fine-tune for 10% of its budget at 0.1x its peak LR; segcons adds SegConsistencyLoss with cpsam (G4's S_train). seg_consistency_weight is a null placeholder until calibrated, which the engines refuse. Predict leaves read last.ckpt. Registers the six tokens in PAPER_KEY and the grouped parser, and admits the pix2pix2d forks to the <80 GB cards like their baseline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dynacell): calibrate the P1 segcons weights seg_consistency_weight per arm is the median over 20 seeded val batches of ||grad base|| / ||grad seg_consistency|| at the fork checkpoint (calibrate_variant.py segcons variant, A40): FNet-3D 0.48 [0.25-0.93], UNeXt2-3D 0.050 [0.024-0.088], pix2pix2d 0.57 [0.32-0.67]. The tool's plain-Dice row reproduces the recorded FNet-3D vscyto3daug 4.5 on its July checkpoint (4.504 [2.58-6.72]). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): reject seg_consistency combined with seg_aux With both terms the logged loss/base_train would be the seg_aux base, not the loss before the consistency term. No arm combines them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-utils): drop unused SegConsistencyLoss options and the 4-D path No arm uses max_samples or the Ctrl-U uncertainty weight, and every engine hands the loss 5-D (B, 1, Z, Y, X) crops (Z = 1 for 2-D models), so the loss now requires 5-D. The z <= 1 early return in nucleus_area_planes is redundant: the guard band of a one-plane stack is plane 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-utils): build the Otsu histogram without a host sync torch.bincount sizes its output on the host, so each focus-plane pick synced the GPU. scatter_add_ into a fixed (rows, 256) histogram does not. Thresholds are bit-identical to the bincount version (new test), and a CUDA sync-debug test pins the no-sync claim; it trips on the old version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(dynacell): simplify CellposeDense normalize99 is hard-coded to 1/99 with float quantiles (no per-call host-to-device tensor), the taper uses BSIZE, checkpointing keys on the tiles' requires_grad alone, and the train() override and requires_grad_ call go: SegConsistencyLoss freezes its segmenter, and the cpsam test now checks that it stays in eval with no grads inside the loss. The net is still put in eval once at construction: cellpose builds it in train mode, whose flows differ from stock inference by up to 1.7 (parity test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynacell): pin SegConsistencyLoss's focus port to the eval nucleus_area_planes / _slab_mip against dynacell.evaluation.focus nucleus_area_plane / slab_mip on 50 blob volumes (Z = 1..40, a constant volume, random scale and offset): planes and slabs (halfwidth 0-2) equal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(dynacell): drop the segcons null-weight placeholder path All three weights are calibrated, so SEG_CONSISTENCY_WEIGHTS is typed float and the placeholder header branch goes. The generator test no longer rebuilds each arm with a null weight (3 extra ViT-L loads); the engine refusal is covered by test_seg_consistency_training. --check: 467 leaves, 0 stale, no leaf changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynacell): skip cellpose tests without cellpose; fix a docstring cellpose is an optional dynacell extra: test_cellpose_dense importorskips it at module top, and the generator's cpsam build test does so in-test so the rest of that module still runs. The gradient test's docstring now says what it asserts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): crop the segcons projection to 240 px, one cellpose tile cellpose get_pad_yx pads a 256-px image to 272, which make_tiles cuts into 2 x 2 tiles: 4x the segmenter compute per sample. A 240-px crop pads to exactly 256, one tile. SEG_CONSISTENCY_ARGS and the loss default move to 240; CellposeDense.tiling exposes the padding/tile layout, and a test pins one tile at the configured crop (and 4 at 256). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): shorten the pix2pix2d fork's EMA half-life Both pix2pix2d arms predict from generator_ema, which starts as the fork checkpoint's shadow. At the baseline's ema_kimg 10, the 1000-step x 16-sample fine-tune leaves 0.5^(16000/10000) = 33% of that start shadow in it; ema_kimg 2 leaves 0.5^8 = 0.4%. Set for both arms of the pair. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): keep CellposeDense in eval under train() Restores the train() override next to the construction-time eval(): the network's train mode changes its outputs (flows off by up to 1.7 vs stock inference), so a parent's train() must not reach it if the wrapper is ever registered as a submodule. Tested through a ModuleDict parent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): reject a non-positive seg_consistency_weight A zero weight builds the segmenter and pays its cost for no term; turning the term off means dropping seg_consistency. The gradient test now checks the step's backbone gradient equals the base gradient plus weight x the term's own gradient, instead of comparing against a zero-weight arm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(dynacell): drop the stale seg_consistency weight placeholder wording The weights are calibrated, so the engine's docstring and error describe a missing weight (no default) rather than a null leaf placeholder, and the generator docstring no longer mentions a null weight. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(dynacell): check every fork-checkpoint tensor in the control build The strict-load test compared only the first state_dict tensor; it now requires every tensor of the built control to equal the fork checkpoint's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(dynacell): recalibrate the segcons weights at the 240-px crop Same protocol at the leaves' one-tile crop (results/*__segcons_crop240): FNet-3D 0.37 [0.14-0.65], UNeXt2-3D 0.032 [0.024-0.066], pix2pix2d 0.19 [0.067-0.34], from 0.48 / 0.050 / 0.57 at the 4-tile 256-px crop. The plain-Dice instrument rows are unchanged to every printed digit. The comment records the convention's known limitation: segaux-style base (pix2pix2d: generator_ema, lambda_l1 * L1, no adversarial term) and no training-batch averaging, kept for comparability with the segaux weights. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(dynacell): keep the global RNG unchanged when building CellposeDense CellposeModel random-initializes its ViT-L before loading the checkpoint, which advanced the global torch RNG between seed_everything and HCSDataModule's torch.randperm train/val split. A segcons arm therefore split its FOVs differently from its same-seed segcons_ctrl, confounding the comparison (measured: seed 42 randperm differs after construction). Construction now runs inside torch.random.fork_rng; a test pins the torch state (it fails on the previous code). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwn9hwgT8uJSoWaCxCAm8J --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

The pix2pix2d overlay copies Run-D's warmup_steps: 8500, an absolute batch count. pix2pix3d fits run 31k-107k batches, so there warmup is 8-27 % of training; the 2D fits run 4.4k-16k, so 5 of 12 (all A549 and iPSC ER) ended still warming up at peak LR, the iPSC fits peaked at 85 % and the joint fits at 53-65 % (wandb lr-AdamW, 2026-10-22). Their ranking against the 3D track is what the paper's 2D appendix compares. pix2pix2d_unetvit_wmatch refits all 12 with one recipe delta, warmup_steps = round(8500 * T2D / T3D) (1177-1328), keeping the 3D schedule shape; _wmatch_seed1 is its second draw for nucleus and membrane (iPSC), matching the existing seed1 arm. New run roots, so the baseline fits, predictions and evals stay. Both names get their own paper keys: canonical_model_name resolves by longest prefix and would otherwise fold them onto pix2pix2d_unetvit's dirs. The leaves keep the baseline's measured 48 GB-card opt-out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

nucleus/membrane_ipsc_trained__2d_focus2d_seed1 score the existing seed1 predictions of the four 2D families that have one (FNet2D, UNeXt2-2D, pix2pix2d, CellDiff-2D; VSCyto2D has none) with the same focus2d recipe, planes and caches as their seed0 buckets, into eval_focus2d_h1/. The paper's 2D appendix uses them as a seed-based ceiling for the 2D-vs-3D family-ranking agreement, in place of the A549 test conditions standing in for replicates. Registered as hand-written: an experiment script copied them from the seed0 buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

Four slab3 predict leaves per pix2pix2d_unetvit_wmatch fit (12) and _wmatch_seed1 fit (2): each is its baseline leaf with the arm's model name, output store, run root and job name, and the checkpoint the fit's own ModelCheckpoint recorded as best (loss/validate_ema, read from last.ckpt: filenames round the loss to 3 decimals and break ties differently). Every fit's log reads `max_epochs=40` reached; warmup peaks at 8-27 % of training, as in the 3D fits, and decays to zero. ER's iPSC-trained baseline has no committed predict leaves (its predictions came from resolved configs in the run root), so those four are derived from the A549-trained ER leaves, which differ only in trained_on, experiment_id, checkpoint, pool path and the job name's A549_ prefix; the same derivation reproduces mito's committed iPSC leaves exactly (4/4). All 56 predictions ran on the plane files' slabs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

…ix2d <org>__2d_focus2d_pix2pix2d_wmatch, one per organelle, hold the pix2pix2d conditions of the three <org>_<pool>__2d_focus2d buckets (and, for nucleus and membrane, of the seed1 bucket) re-pointed at the pix2pix2d_unetvit_wmatch[_seed1] stores, pred caches and eval dirs: 56 conditions with the same focus2d recipe, planes and caches as the baseline buckets, whose top-level blocks are identical across pools. Registered as hand-written (an experiment script writes them). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MzpgExfYMKCgogfnhcSgDc

…#531) * perf(viscy-data,viscy-utils): read and write each predict volume once Stride-1 depth windows made predict re-read every input plane once per window covering it, and the writer did a zarr read-modify-write of the whole window for every batch. On one H200 that I/O was 76% of FNet3D's end-to-end predict time (14.3 s/volume against 3.4 s of forward) and 96% of UNeXt2's. HCSDataModule.predict_read_once (default True) now has SlidingWindowDataset cut every window from one cached (position, timepoint) volume. HCSPredictionWriter assembles the volume in memory with the same blend/center arithmetic and writes it once, when its last window arrives or at predict end. The stored bytes and completion markers are identical to per-window I/O; the old path stays behind predict_read_once=False and is still used with predict_z_planes, write_input and target_2d padding. The flag is loading-only for the prediction settings hash. A new integration test checks bitwise identity against per-window I/O for blend and center, batch 1 and 3, and a partial run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * refactor(viscy-data,viscy-utils): drop predict_read_once and buffer every write mode The flag only kept the per-window path reachable, and the datamodule and the writer each decided eligibility from it. The dataset now caches whenever no plane selection is set, and the writer buffers each (position, t) volume in every mode except write_input, including target_2d padding and predict_z_planes, which store the same bytes buffered. One read/write target replaces the paired branches in write_sample. The settings-hash exclusion for the flag goes with it. Tests compare the buffered writer against a per-window reference writer for blend, center, target_2d and z_planes, including partial volumes flushed at predict end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * fix(viscy-utils): buffer predictions in the dtype of an existing output array The per-window path writes into an existing array (overwrite) and casts to its dtype, so later windows blend against stored values. The buffer took the prediction's dtype instead: a float16 predict over a float32 store blended in float16 and stored other bytes. The buffer now takes the existing array's dtype, as the per-window writes would. The test overwrites a float32 store with float16 predictions through both writers; it fails without the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * perf(viscy-data): read the cached predict volume with per-channel basic slices The whole-volume read used oindex, whose coordinate indexer peaks at ~7x the output size; HCSDataModule's mmap preload moved off it for the same reason (6ec0d6f). Per-channel basic slices read the same bytes. The float32 cast no longer copies a volume that is already float32. cache_volume now refuses fg_mask_key, since its single slot cannot serve image and mask reads, and its docstring states that each DataLoader worker keeps its own copy. A test checks that windows cut from the cached volume equal windows read one by one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * fix(viscy-utils): flush only the planes a buffered volume's windows reached The buffered flush wrote planes [0, stop) from a zero-filled buffer, so planes no window predicts (target_2d padding, unselected planes) were overwritten with zeros. The per-window path never touched them: a target_2d run over a 3D store kept the earlier planes there, the buffered one zeroed them (measured in review). The buffer now records the first and last plane its windows reach, and the flush writes that range only. The test overwrites a 3D store with a target_2d run through both writers; it fails with the old flush range. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * fix(viscy-utils): grow the buffered volume for predictions deeper than their window The per-window path resized the zarr array whenever a window's planes ran past it. The buffer was fixed at the source depth, so a model that returns more planes than its input window failed to broadcast. The buffer now grows the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * perf(viscy-data): cache the predict volume only without loader workers Loader workers each take every Nth batch, so with workers each would read and hold the whole volume: more reads than per-window I/O for short windows, and one volume of RAM per worker. HCSDataModule now caches only with num_workers == 0, as every dynacell predict leaf sets. The writer still assembles volumes in memory either way. The equivalence test's reference now reads per window (num_workers=1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * fix(viscy-data): key the cached predict volume on the array object The key used the array's plate-relative path, which positions of different plates share, so a dataset over two plates could serve one plate's cached volume for the other's windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * perf(viscy-data): fill the cached predict volume in place Concatenating per-channel reads and then casting to float32 held the per-channel arrays, the concatenation and the cast at once, about 3x the volume for uint16 sources. One preallocated float32 array filled channel by channel keeps the peak at its size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * fix(viscy-utils): keep plane-restricted predicts on per-window writes A plane selection writes single planes, and the buffered flush writes the range from the first to the last one, so a noncontiguous selection ([0, 2]) wrote the zero buffer over the unselected planes between them, which per-window writes leave as stored, and a sparse one ([0, 999]) wrote every plane in between. Each window is a single plane with no blending, so buffering saved nothing there. Plane-restricted predicts now write per window; the equivalence test asserts that. Reported by Copilot on #531. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG * perf(viscy-data): release the cached predict volume before reading the next The previous volume stayed referenced until its replacement was read, so two volumes were resident at each (position, t) change. Windows own copies, so the old cache can go first. Reported by Copilot on #531. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01APoGhoKfkCho9Z1VMS5jsG --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

… async in predict (#532) * perf(viscy-utils): blend predict volumes on the device, copy once After the read-once change (#531) the FNet3D predict loop still spent ~1.5 s per volume on the host: a float64 numpy blend of every 32-plane window and a device-to-host copy per window. The writer's volume buffer now lives on the prediction's device in the store dtype. Each window blends there in float64 with _blend_in's operation order (separate multiply, divide and add; the factors are a device tensor, so CUDA divides instead of multiplying by a reciprocal) and rounds to the buffer dtype once, as the numpy blend and zarr write did. The volume reaches the host once, at its flush. Torch rounds float64 to float16 through float32, so non-float32/64 buffers are cast by numpy to keep single rounding. The integration test now runs the per-window reference against the buffered writer on the CPU and on a GPU when present, plus a float16 store. Mutations caught: a float32 device blend fails all 8 blend cases on both devices; a reciprocal-multiply blend fails the 4 full-run blend cases; a torch float64->float16 cast fails the new rounding test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-data): normalize a cached predict volume once With cache_volume the dataset still ran NormalizeSampled on every depth window, so each plane was normalized once per window covering it (32 times for FNet3D), part of the ~0.5 s per volume of per-window host work left after #531. NormalizeSampled is now marked pointwise: each voxel maps through precomputed statistics alone. When every predict transform is pointwise, the dataset normalizes the cached (position, timepoint) volume once, with that timepoint's statistics, and cuts the windows from it without transforming or copying them again (collate copies). Any other transform keeps running per window. The new test compares cached and per-window samples bytewise with FOV and timepoint statistics that differ per FOV and frame, and with a window-mean transform that is not pointwise. Mutations caught: dropping the pointwise check fails the window-mean case, frame-0 statistics fail the timepoint case, skipping the volume transform fails both pointwise cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-utils): write finished predict volumes on a background thread Each finished volume's zarr write (~0.4 s per volume for FNet3D on the A40) blocked the predict loop. The writer now copies the volume to the host and hands the write to one background thread, so it overlaps the next volume's forward passes. Ordering and failure stay as before: - writes run one at a time, in submission order, and at most one is in flight, which bounds the host copies held to two volumes; - a FOV's completion marker is written on that thread after its last volume's write returns, never before; - a write error is raised at the next batch, at the next flush, or at predict end, and its FOV is never marked complete; - the store dtypes the buffers follow are read at predict start, so the main thread no longer touches the store while writes run. Tests: a writer that holds each write for 0.2 s checks that every marker sees all of its FOV's voxels and is written off the main thread; failing the first FOV's last volume stops the run before the next FOV, and failing the last volume fails the predict at its end. Mutations caught: marking on the main thread after queueing the write fails the ordering and both failure tests; dropping the re-raise at predict end fails the last-volume case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-utils): cache the device blend factors _blend_in built its factor tensor from a Python list on every window. On a GPU that is a pageable host-to-device copy, which waits for the window's forward pass, so the host could not run ahead into the next window's load. The factors depend only on (depth, samples), so they are now built once per value, dtype and device and reused; the values and the blend's operation order are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-utils): read store dtypes only for a pre-existing store The dtype lookup for each source array ran even on a fresh store, where every lookup misses. It now runs only when the store existed and keeps torch dtypes, so _buffer_volume takes them directly. _write_volume derives the array shape from its planes instead of taking it as an argument. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-data): decide per-window transforms once One _transform_windows flag, set at construction, replaces the repeated "transform and not transformed per volume" test at the window copy and the transform call; the redundant transform-is-None clause goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-transforms): mark MinMaxSampled pointwise MinMaxSampled clamps and rescales each voxel with precomputed range statistics, so it is pointwise like NormalizeSampled, and predict configs using it can normalize a cached volume once too. The cached vs per-window bytes test gains a MinMaxSampled case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(viscy-utils): reuse the shared data module in the writer tests The float16 and marker-ordering tests built the same HCSDataModule _ones_data_module returns; the helper now takes batch_size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-utils): skip the float64 copies for a volume's first window _blend_in stores the first window of a volume unchanged, so promoting it and the buffer slice to float64 on the device only cost two window-sized float64 allocations. For a model whose single window covers the whole volume (CELL-Diff at z_window_size 48 on A549), the device buffer is then the writer's only added GPU memory: one volume, 118 MB at 48x640x960 float32. The stored value is the same either way, since float32 -> float64 -> float32 is exact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): cache blend factors as normal tensors The cached device factors were built inside predict's inference mode, so they were inference tensors. A later grad-enabled _blend_in caller (cytoland's in-memory sliding window outside no_grad) would fail with "Inference tensors cannot be saved for backward". They are now built with inference mode off. The new test caches the factors under inference mode and then blends with gradients; it fails without the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): close the store when a predict-end write fails on_predict_end re-raised a failed background write before shutting the flush pool down and closing the plate, so a failed last write left the store open. The flushes now run in a try whose finally waits for the last write and closes the store; the write error is raised after. The failing-write test now checks the store was closed and the pool shut down when the error surfaces at predict end; it fails without the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-data): refuse nonzero-fraction retries with the volume cache min_nonzero_fraction replaces a failing window with a random one, which breaks the in-order access the volume cache relies on. Once the cached volume is normalized once, the check would also judge normalized rather than raw values. HCSDataModule never passes the filter to the predict dataset (it is train-only), so no current run combines them; the dataset now refuses the combination instead of silently changing which windows it serves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): refuse a predict window that arrives twice The writer assumes each (position, t, z) window arrives once, in order. A repeat of a window whose volume was already flushed would blend against a fresh zero buffer, overwrite the stored planes with that partial result and mark the FOV complete again; a repeat inside an open volume would blend twice. Both now raise. The new test feeds a volume's first window again after the volume; it fails without the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(viscy-data): skip the volume transform for an empty Compose all() over no transforms is True, so normalizations=[] pushed every cached volume through a no-op Compose and copied it back channel by channel. An empty transform now counts as nothing to apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(viscy-utils): spell out the pre-existing store dtype read A plain "if existed:" guard and a try/except around the lookup alone replace the conditional iterable and the bare pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(viscy-utils): close the store when predict fails mid-run A volume write that fails on the flush thread surfaces at the next batch or flush and aborts predict before on_predict_end, so the flush pool and the output store were left open. The writer now implements on_exception: it lets the last write finish and closes the store. The same cleanup ends on_predict_end. It is a no-op for whatever setup did not reach (plate and pool start as None), so a failure inside on_predict_start is safe, and a second call is harmless. The failing-write test now asserts the store is closed and the pool shut down when the failure stops the run mid-predict (volume 2) as well as at predict end (volume 4); the mid-run case failed before this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

alxndrkalinin · 2026-10-27T01:41:00Z

Closing in favor of a three-PR split of the same changes, so the shared-package changes and the DynaCell application can land on main without the Spotlight experiments:

Port shared VisCy package changes from DynaCell v1.1 #533: shared packages (base main). Covers viscy-utils, viscy-data, viscy-models and viscy-transforms (including the segmentation-aware losses and fg-mask plumbing), plus root build/CI and the cytoland/dynaclr fixes.
DynaCell v1.1 application #534: applications/dynacell (stacked on Port shared VisCy package changes from DynaCell v1.1 #533). The benchmark, evaluation pipeline, launcher, tools and configs, without the Spotlight experiment configs.
Spotlight v2 segmentation-aware-loss #535: Spotlight v2 (stacked on DynaCell v1.1 application #534). Only adds files: the segmentation-aware-loss configs, their eval configs, and the generators and tools for them.

Stacked together, #535's head reproduces dynacell-models @ 8e74a7eb except for 24 intentional files. They are: the CI workflows; the tutorials' install lines (viscy-data from GitHub, since the PyPI wheel lacks the new predict API); the eval-config generator's handling of model dirs from other campaigns (skip known Spotlight arms, raise on unknown dirs), with its tests; and review fixes in viscy-utils (health checks, CLI checkpoint setup, CLAHE zero-range guard, lazy callback imports) and the instance-AP coverage audit. The per-file list is below.

git diff --stat origin/dynacell-models #535-head

 .github/workflows/lint.yml                         |   2 +-
 .github/workflows/test.yml                         |  38 ++++++--
 .../cytoland/examples/vcp_tutorials/hek293t.ipynb  |   6 +-
 .../cytoland/examples/vcp_tutorials/hek293t.py     |   6 +-
 .../examples/vcp_tutorials/neuromast.ipynb         |   6 +-
 .../cytoland/examples/vcp_tutorials/neuromast.py   |   6 +-
 .../examples/vcp_tutorials/quick_start.ipynb       |   8 +-
 .../cytoland/examples/vcp_tutorials/quick_start.py |   8 +-
 .../tools/generate_grouped_eval_configs.py         |  53 +++++++++--
 .../tools/generate_grouped_eval_configs_test.py    |  46 +++++++++
 .../tools/generate_instance_ap_eval_configs.py     |  10 +-
 .../generate_instance_ap_eval_configs_test.py      |  16 ++++
 packages/viscy-utils/pyproject.toml                |   1 +
 .../src/viscy_utils/callbacks/__init__.py          |  45 +++++----
 .../src/viscy_utils/callbacks/monitor_health.py    |   8 +-
 .../viscy_utils/callbacks/monitor_health_test.py   |  31 +++++++
 .../src/viscy_utils/callbacks/optimizer_health.py  |  16 ++--
 .../viscy_utils/callbacks/optimizer_health_test.py |   8 ++
 packages/viscy-utils/src/viscy_utils/cli.py        | 103 +++++++++++----------
 packages/viscy-utils/src/viscy_utils/meta_utils.py |   6 +-
 packages/viscy-utils/src/viscy_utils/trainer.py    |   5 +-
 packages/viscy-utils/tests/test_lazy_imports.py    |  19 ++++
 packages/viscy-utils/tests/test_meta_utils.py      |  10 ++
 uv.lock                                            |   2 +
 24 files changed, 341 insertions(+), 118 deletions(-)

Checks:

Port shared VisCy package changes from DynaCell v1.1 #533: CI green on 0b1cb979 (29 checks passed, 1 skipped).
DynaCell v1.1 application #534 / Spotlight v2 segmentation-aware-loss #535: local full suites green on a04b3eb5 / 430f521c (dynacell 4089 passed, 0 failed; dynacell 5047 passed, 0 failed). GitHub CI runs on them once they retarget to main.

dynacell-models is not deleted. It stays as the paper-cited snapshot of DynaCell v1.1.

🤖 Generated with Claude Code

alxndrkalinin and others added 30 commits August 3, 2026 14:03

docs(dynacell): document benchmark validation split

8fd2695

docs(dynacell): align overview with camera-ready benchmark

22cec5a

alxndrkalinin and others added 25 commits October 13, 2026 13:04

Merge branch 'dynacell-models' of https://github.com/mehta-lab/VisCy …

cb981ce

…into dynacell-spotlight-v2-eval-waves

Merge branch 'dynacell-models' of https://github.com/mehta-lab/VisCy …

7a224b8

…into dynacell-spotlight-v2-eval-waves

This was referenced Oct 26, 2026

Port shared VisCy package changes from DynaCell v1.1 #533

Merged

DynaCell v1.1 application #534

Merged

Spotlight v2 segmentation-aware-loss #535

Open

alxndrkalinin closed this Oct 27, 2026

Navigation Menu

Repository navigation

DynaCell v1.1 - #494

DynaCell v1.1#494
alxndrkalinin wants to merge 788 commits into
mainfrom
dynacell-models

alxndrkalinin commented Jul 29, 2026

Uh oh!

alxndrkalinin commented Oct 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

alxndrkalinin commented Jul 29, 2026

Uh oh!

alxndrkalinin commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alxndrkalinin commented Oct 27, 2026 •

edited

Loading