feat(pyramid): pyramids on images and label images; mantis-v2 adds them after tracking and QC (#359) by ieivanov · Pull Request #430 · Biohub/biahub

ieivanov · 2026-10-21T00:36:23Z

Closes #359. Part of #363. Builds on #413 (feat/track-labels-geff), which it is based on until that merges; paired iohub PRs: Biohub/iohub#488 and, stacked on it, Biohub/iohub#461; with #413's Biohub/iohub#489 they are combined on iohub's dev/pyramid-shards (pinned here at 82808fd).

What changes

biahub pyramid is rewritten to the processing-command template and computes pyramids on images and label images of each position, in place:

--levels (default 3, was 4: the DCA v0.2 target), --method for the images (mean), and a new --label-method for every labels/<name> (max; max/stride/mode only, so no level holds a label id level 0 does not: DCA's "maximum filter or decimation filter for label arrays").
Spatial axes of size 1 are not downsampled: the projected (Z=1) plate gets factor 2 in Y and X and 1 in Z, with Z's scale left alone. Time and channel are never downsampled. Every level keeps the level-0 chunk shape, clamped to the level; shards are sized per level to the level-0 shard's byte size (iohub#461), so the assembled plate's shards stay about 4.5 GiB at level 1 and the projected plate's level-1 shards grow along time instead of shrinking with the frame.
Layered as _pyramid_position → pyramid() → pyramid_cli; --cluster slurm|local|debug, --monitor, --init replace --local. --init checks the first position against the requested depth (a plate with a pyramid of another depth is refused up front) and prints the RESOURCES line; it scaffolds nothing. Rerunning recomputes the same levels, so retries are idempotent.
PyramidSettings (settings.py) validates the options and is the provenance record, biahub-pyramid: {levels, method, label_method, dims} on each position.
Partition preempted, slurm_output/ beside the plate, the three-example docstring. Audited against the biahub-cli-consistency skill: conforms, with the deviations inherent to an in-place command (no -c/-o, no _init_output_plate, no @resume()), the same as track.

iohub (Biohub/iohub#488). Position and PositionLabel share a new MultiscaleNode base, so PositionLabel.compute_pyramid(levels, method, dims) exists and the label levels inherit the level-0 scale, chunks and shards like the image's (the old PositionLabel.initialize_pyramid wrote [1, 2, 2, 2] scales and dropped shards). iohub compute-pyramid --label-method added. Also fixes downsample_block(method="mode"), which returned half-sized, scrambled levels.

mantis-v2 pipeline.

New nextflow/modules/pyramid.nf (init + run subworkflows), included twice under aliases for the assembled and the projected plates.
New --pyramid_levels (default 3; 1 turns the step off; moot without --concatenate_config). Both inits run in the init phase once their plate is scaffolded.
Pyramids run after the last step that writes pixels or labels into each plate (the last of segment and track where they write there, else assemble / project). QC is now the very last step: its compute is gated on the pyramids (or on tracking / the creating step when pyramids are off) instead of running concurrently with projection and tracking, so imaging-qc's stamps on the position attrs never race another writer of the same zarr.json and the report describes the finished plates. Level 0 is never touched. Both pyramid dones join the cleanup gate and the Slack step list (pyramid (assembled), pyramid (projected), then QC).
Docs: README, the run template, the reconstruct-with-nextflow skill's step tables, and a caveat on rerunning pyramids after a partial relaunch.

Configs stop passing shards_ratio. ConcatenateSettings and ProjectionSettings take shards in iohub#461's forms (a size target such as 5GB, XY/XYZ, or an explicit TCZYX shape in elements); shards_ratio is refused with a message naming the new key. Concatenate reads the shard's T extent back from the plate its init created to size its workers, and the tracked label image copies the dtype, chunks and shards of the label image it is linked from (segment's; ids beyond that dtype are refused); segment's own labels take LABEL_CHUNKS (16 × 1 × 128 × 128) and a 200MB size target, one time chunk by the full plane on these plates. Templates: a549/concatenate.yml and zebrafish/concatenate.yml move to chunks_czyx: [1, 8, 256, 256] (2 MiB float32 chunks) and say shards: 5GB (A549: 6 timepoints by the full volume, 4.5 GiB; dynatrack: 9 timepoints, 4.98 GB, under the DCA 5 GB SHOULD), a549/project.yml says shards: 200MB (21 frames by the full frame, 193 MB, instead of 16). Caveat §7 of the pipeline skill is rewritten around size targets.

Testing

tests/test_cli/test_pyramid_cli.py (new): image and label levels/shapes/scales, label id sets preserved, Z=1 plates downsampled in YX only, rerun recomputes / different depth refused, --init RESOURCES and no writes, option validation. Full pytest tests with the pinned iohub (ultrack-dependent test_track_cli.py deselected, as in feat(segment,track)!: segment and track write into the plate; labels and tracks.geff (#360, #361) #413's runs; nothing here touches track): 631 passed, 13 skipped. make lint clean; nextflow lint clean on mantis-v2.nf and modules/pyramid.nf.
Real data: two FOVs of 2026_08_11_A549_SEC61B_DENV_debug/v8/4-assemble/<ds>_proj.zarr (67 × 6 × 1 × 1664 × 1193 float32 plus labels/nuclei_prediction), biahub pyramid --cluster debug --levels 3: ~55 s per position in-process; levels 832 × 597 and 416 × 299 on image and labels, chunks/shards halved, label id sets identical at every level (19 and 35 ids in the first 5 timepoints, no new ids), Z scale unchanged, biahub-pyramid record written. The resources line requested 4 CPUs / 16 GB / 30 min.

Follow-ups

Repoint the iohub pin at main once feat(ngff): compute label pyramids with the image's multiscale machinery iohub#488 and docs(repo): meet CZI open source requirements #461 merge (order: #489, #488, then docs(repo): meet CZI open source requirements #461 rebased on main).
biahub concatenate/segment do not write pyramids; the pipeline adds them at the end, so curated datasets built with concatenate need a biahub pyramid pass.

🤖 Generated with Claude Code

Tracking no longer creates a separate 5-track/<ds>.zarr plate holding the labels as an image channel. It writes into each -i position of the assembled plate (#360, #361): - labels/<target_channel> as an OME-NGFF 0.5 label image: uint32, TZYX with the image's full Z, chunks (16,1,128,128) (1 MB, the DCA SHOULD), shards of 16 timepoints by the YX plane. 2D masks go in the middle plane of the tracked z-window (middle of the volume for a full-stack projection, the plane itself for a one-plane window, middle of a projected section); 3D masks fill the window. The track settings and z placement are recorded under `biahub-track` on the label image. - tracks.geff via ultrack's Tracker.to_geff (zarr v3, seg_id = track_id, related_objects pointing at the labels, axes with units and scale). - tracks_<row>_<col>_<fov>.csv, kept for downstream readers for now. `track --init` creates the empty label images with create_empty_plate(label_names=...). -o is now a working directory for the Ultrack databases and SLURM logs. TrackingSettings drops output_ome_zarr_version, since the labels take the input store's version. Nextflow: --qc_track_config and qc_track.yaml are removed. a549/qc.yaml counts instances in the label image (imaging-qc#243, `label:`), so QC is planned after track --init and computed after tracking. Pins: iohub dev/dca-labels (iohub#485, #486), the ieivanov/ultrack fork (royerlab/ultrack#288), imaging-qc f5ae656, geff/geff-spec >= 1.3. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…abels field Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

… fix Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Each worker now builds its Ultrack database in a fresh scratch directory under its own $TMPDIR and deletes it once the FOV is exported. `-o` becomes optional and keeps the databases in {-o}/{row}_{col}_{fov} when given (e.g. to re-solve with other weights); the SLURM logs go to {-o}/slurm_output, else ./slurm_output, since submitit's folder must be on a shared filesystem. The scratch directory also holds ultrack's own temporary zarr arrays: create_zarr backs its default store with a TemporaryDirectory it lets go out of scope, so those directories were recreated by the first write and left in $TMPDIR, which on Bruno outlives the job. The full resolved Ultrack config (defaults included) is recorded under `biahub-track.ultrack_config` on each label image, so provenance no longer depends on config.toml beside the database. The worker returns nothing rather than a label array inside the deleted scratch directory. Nextflow passes no -o, so tracking has no step directory (no 5-track/, no README, no chmod of the SQLite file) and takes no number; its real logs stay in nextflow/slurm_output/track/. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

create_zarr now deletes its temporary store with the array, so the per-FOV temporary directory only needs to hold the Ultrack database again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Add `biahub project`, which turns a 3D plate into a 2D (Z=1) plate with one projection per output channel: max, mean, sum over a Z range or a window around a plane, or one plane (`slice`). The plane may be fixed or the in-focus plane found with the registration engine's focus criterion (FocusSettings subclass: centre crop, NA, wavelength), once per position (median over T) or per timepoint. Ranges are half-open like concatenate's Z_slice and relative to `center` when one is set, so the A549 60-plane window is `center: focus, range: [-20, 40]`. Unlisted input channels are dropped with a warning. The focus plane of every timepoint is recorded in the output position's `biahub-project` metadata. Segment and track stop selecting and projecting Z-planes. Segment loses `z_slice_2D`: a model without do_3D segments in 2D and needs a Z=1 input. Track loses `z_slicing` and the focus window; `output_mode` is checked against the input Z. Both raise, pointing at `biahub project`, when a 3D input meets a 2D request. Old keys are rejected with a migration message. Nextflow: a project step between assemble and track (`5-project`, `6-track` for A549); track reads the projected plate when project runs. The A549 project config reproduces the previous tracking input (mean over the 60-plane focus window on Phase3D), measured on the 800x800 centre crop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ill checks it The A549 projection config lists every channel of a typical assembled plate under the pre-rename names project sees inside the pipeline: the in-focus plane of BF and Phase3D, the full-stack max of the raw fluorescence channels, and the mean of the virtual-stain predictions over the 60-plane focus window. The focus is found on Phase3D per timepoint, as biahub#400 asks for inference. The brightfield key is `BF`, matching reconstruct.yml; the fluorescence names and an older `BF - Oblique` are per-dataset edits. The reconstruct-with-nextflow check script also validates project.yml both ways (a listed channel the plate lacks is a MISS, an assembled channel the config omits is NOT PROJECTED) and track.yml against the projected plate. The skill carries unprojected channels into the pre-run plan for the user to decide, runs the rename on the projected plate too, and suggests the `BF - Oblique` edit in both reconstruct.yml and project.yml when the raw store names it so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…4 accumulation NumPy already accumulates safely (float64 for integer input, pairwise float32 for float32 input), so mean and sum no longer force a float64 temporary. The output plate is float32 whenever a channel uses max, mean or sum -- float32 in and out is the common case -- and keeps the input dtype only when every channel is a slice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The focus record under the output position's biahub-project metadata is focus: {channel, per, z, z_per_t}; the focus_ prefix repeated the block name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

`range` meant absolute planes without a center and offsets with one. The two forms now have their own keys: `start`/`stop` are absolute, half-open like concatenate's time_indices; `below`/`above` count planes around `center`, selecting [center - below, center + above), so the A549 window is `center: focus, below: 20, above: 40`. The validator refuses mixing them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The projected store is a deliverable like the assembled plate: cleanup removes only its slurm_output/ and .iohub-progress/, as the wiring already did; say so in the comment. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… shards ProjectionSettings takes the output chunk shape and shards ratio with concatenate's names and semantics (shard = chunks * ratio, C not sharded) and passes them to create_empty_plate; None keeps iohub's defaults. The A549 config, following the Dynamic Cell Atlas array standard where a 2D float32 plate can: one-frame 256 x 256 chunks, the grid the assembled and tracking stores share, and shards of 16 timepoints by the full frame (the standard's T-shard floor; ~150 MB each, 30 files per position instead of 400). iohub writes one shard's timepoints per unit, so resume is 16 frames coarse. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The focus pre-pass ran the timepoints one after another while the job's other workers idled; with per-timepoint focus on a long timelapse it was the serial part of the step. The timepoints are independent, so with more than one worker they now run in a spawned ProcessPoolExecutor sized by the job's num_workers (the count the projection itself uses), each worker reading its own volume. One worker keeps the in-process loop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…>_proj.zarr The projected plate is the 2D companion of the assembled plate, not a step of its own: the pipeline now writes it to <N>-assemble/<DATASET>_proj.zarr, so project has no step directory and track stays 5-track. Cleanup already covers the shared directory's slurm_output/ and .iohub-progress/; the slurm_output README notes that a directory can hold two steps' stores. The skill docs, run template and config comments follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

iohub reads one timepoint's (C_read, Z, Y, X) block per worker, so the request must grow with the listed channels: the fixed ram_multiplier of 4 gave 4 GB/CPU whether a config read 2 channels or 6, and the A549 config (6 channels, 3.8 GiB per read, ~5-6 GiB peak) would have hit the cgroup. ram_multiplier is now len(read_indices) + 2 (the read, the window copy and the shard's output buffer): 6 GB/CPU for A549, unchanged for 2 channels. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Follows c8f8e78, which dumps the focus settings (minus axes) into the biahub-project focus record next to z and z_per_t. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…ition One position of 2026_08_11 SEC61B v8 (6 channels, 86 x 1664 x 1193 float32, 5 shard units busy) peaked at 22 GB max RSS / 28 GB over the process tree, ~5-6 GiB per worker against the 6 GB/CPU requested. +4 volumes requests 7 GB/CPU there, a margin for a timelapse that keeps all 8 workers busy. The time request (70 min for a 3.6 min run) is left as is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…k reads With `biahub project` (#424) 2D tracking reads the projected (Z=1) plate, so the labels, tracks.geff and the tracks CSV are written into each position of that plate, and the label image takes the plate's Z. The plane-placement code (tracked_z_window, write_tracking_labels' z window, the z_window/z_index provenance) goes away; 3D tracking without project writes into the assembled plate as before. The sibling filter (#415) runs before the GEFF export, which now passes the filtered tracks table to ultrack's `to_geff(tracks_df=...)` (royerlab/ultrack#290, pinned to the ieivanov fork until it merges), so the GEFF, the labels and the CSV agree. Nextflow: track reads and writes the projected plate when project runs; the instance-count QC moves from qc.yaml to a new --qc_project_config (a549/qc_project.yaml) on the projected plate, planned after track's init and computed after tracking; qc.yaml is pixel metrics on the assembled store again. The skill's check script and docs follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

TrackingSettings.segmentation_method defaulted to foreground_contour, the legacy path no production config has used since April; a config that omitted the key fell back to it silently. The default is now cellpose, and cellpose without a cellpose_config is refused at load time instead of falling back. The unit tests, which need no weights or GPU, name foreground_contour explicitly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

run_pyramid requested the init estimate verbatim on every attempt, so an estimate that was too low failed all five attempts of every position with the same 137 and aborted the run. The v9 debug run of 2026_08_11_A549_SEC61B_DENV asked 16 GB 25 times and was OOM-killed 25 times. Use retry_memory like the QC and virtual-stain processes: the next attempt doubles after a 137 and keeps the same request after a preemption. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

run_project asked for its init estimate verbatim on every retry, so a low estimate failed all attempts with the same 137. Use retry_memory like the other SLURM steps (fix/nextflow-retry-memory-cap carries the 244 GB cap). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

aofei-liu

i think the description sounds good, idk if merging is blocked on approval so i'll approve for now! I think the only thing to check is to make sure that the shard > 1GB when array > 1GB rule is satisfied per FOV

(ignore the original message, meant to put that on another PR)

… 54 positions The v9 debug run of 2026_08_11_A549_SEC61B_DENV measured every position of the assembled plate (level-0 shards 4.51 GiB): 46.5 GB peak RSS each (10.3x the shard) and up to 67.3 min, the slowest on the 120 GB cpu-c nodes. The old 12x / 0.15 asked 56 GB / 70 min, 4% above the slowest position. 14x the shard and 0.3 min per volume ask 64 GB / 130 min there; the projected plate keeps the 16 GB floor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… float32 max picks an existing voxel, so promoting a max-only plate to float32 doubled its storage for nothing and would truncate uint32 labels above 2^24. A store has one dtype, so a plate with any mean/sum channel is still float32 throughout and its max/slice channels are widened on write. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

One pin for the whole chain: main now carries #489 (output_label, create_empty_labels), #488, #461 and #491, so the 16cdc4f branch pin and the dev/pyramid-shards combination branch retire. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

A label image now has the image's axes with one channel rather than TZYX: the DCA array standard wants every array 5D, and OME-NGFF's own layout note reads a label's dimensions against the image's. iohub creates and writes them that way (czbiohub-sf/iohub, label channel axis); here the DCA label chunks become (16, 1, 1, 128, 128), segment's shape check and track's label reads and writes index the channel axis, and the tests and skill notes follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

`label_layout` passes iohub `label_shards="200MB"` (Biohub/iohub#461: the largest chunk-aligned shard under the target, filled X, Y, Z then T, C never grown) instead of a computed `label_shards_ratio`, which iohub has deprecated. On the A549 projected plate the result is the same geometry as before, one 16-timepoint chunk by the full frame; #430's segment labels use the same constant, so the two branches agree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… pin 0cb3e35 label_layout keeps source_label (the tracked labels copy the segment labels' dtype, chunks and shards) on the 5D layout; the pyramid tests create and expect (T, 1, Z, Y, X) label levels.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…and tracks.geff (#360, #361) (#413) * feat(track): write labels and tracks.geff into the assembled plate Tracking no longer creates a separate 5-track/<ds>.zarr plate holding the labels as an image channel. It writes into each -i position of the assembled plate (#360, #361): - labels/<target_channel> as an OME-NGFF 0.5 label image: uint32, TZYX with the image's full Z, chunks (16,1,128,128) (1 MB, the DCA SHOULD), shards of 16 timepoints by the YX plane. 2D masks go in the middle plane of the tracked z-window (middle of the volume for a full-stack projection, the plane itself for a one-plane window, middle of a projected section); 3D masks fill the window. The track settings and z placement are recorded under `biahub-track` on the label image. - tracks.geff via ultrack's Tracker.to_geff (zarr v3, seg_id = track_id, related_objects pointing at the labels, axes with units and scale). - tracks_<row>_<col>_<fov>.csv, kept for downstream readers for now. `track --init` creates the empty label images with create_empty_plate(label_names=...). -o is now a working directory for the Ultrack databases and SLURM logs. TrackingSettings drops output_ome_zarr_version, since the labels take the input store's version. Nextflow: --qc_track_config and qc_track.yaml are removed. a549/qc.yaml counts instances in the label image (imaging-qc#243, `label:`), so QC is planned after track --init and computed after tracking. Pins: iohub dev/dca-labels (iohub#485, #486), the ieivanov/ultrack fork (royerlab/ultrack#288), imaging-qc f5ae656, geff/geff-spec >= 1.3. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): repin iohub dev/dca-labels after #485 drops its legacy labels field Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): repin iohub dev/dca-labels after iohub#486 label-default fix Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(track): keep no Ultrack databases by default Each worker now builds its Ultrack database in a fresh scratch directory under its own $TMPDIR and deletes it once the FOV is exported. `-o` becomes optional and keeps the databases in {-o}/{row}_{col}_{fov} when given (e.g. to re-solve with other weights); the SLURM logs go to {-o}/slurm_output, else ./slurm_output, since submitit's folder must be on a shared filesystem. The scratch directory also holds ultrack's own temporary zarr arrays: create_zarr backs its default store with a TemporaryDirectory it lets go out of scope, so those directories were recreated by the first write and left in $TMPDIR, which on Bruno outlives the job. The full resolved Ultrack config (defaults included) is recorded under `biahub-track.ultrack_config` on each label image, so provenance no longer depends on config.toml beside the database. The worker returns nothing rather than a label array inside the deleted scratch directory. Nextflow passes no -o, so tracking has no step directory (no 5-track/, no README, no chmod of the SQLite file) and takes no number; its real logs stay in nextflow/slurm_output/track/. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): pin iohub to iohub#486 now that #485 is on main Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): repin iohub#486 after per-position label defaults fix Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): repin iohub#486 after review fixes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): pin ultrack to royerlab main now that #288 and #289 merged Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * refactor(track): drop the tempfile redirect now that ultrack#289 merged create_zarr now deletes its temporary store with the array, so the per-FOV temporary directory only needs to hold the Ultrack database again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): pin iohub to main now that #486 merged Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(project)!: biahub project CLI; segment and track no longer project Add `biahub project`, which turns a 3D plate into a 2D (Z=1) plate with one projection per output channel: max, mean, sum over a Z range or a window around a plane, or one plane (`slice`). The plane may be fixed or the in-focus plane found with the registration engine's focus criterion (FocusSettings subclass: centre crop, NA, wavelength), once per position (median over T) or per timepoint. Ranges are half-open like concatenate's Z_slice and relative to `center` when one is set, so the A549 60-plane window is `center: focus, range: [-20, 40]`. Unlisted input channels are dropped with a warning. The focus plane of every timepoint is recorded in the output position's `biahub-project` metadata. Segment and track stop selecting and projecting Z-planes. Segment loses `z_slice_2D`: a model without do_3D segments in 2D and needs a Z=1 input. Track loses `z_slicing` and the focus window; `output_mode` is checked against the input Z. Both raise, pointing at `biahub project`, when a 3D input meets a 2D request. Old keys are rejected with a migration message. Nextflow: a project step between assemble and track (`5-project`, `6-track` for A549); track reads the projected plate when project runs. The A549 project config reproduces the previous tracking input (mean over the 60-plane focus window on Phase3D), measured on the 800x800 centre crop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(nextflow): A549 project.yml projects every assembled channel; skill checks it The A549 projection config lists every channel of a typical assembled plate under the pre-rename names project sees inside the pipeline: the in-focus plane of BF and Phase3D, the full-stack max of the raw fluorescence channels, and the mean of the virtual-stain predictions over the 60-plane focus window. The focus is found on Phase3D per timepoint, as biahub#400 asks for inference. The brightfield key is `BF`, matching reconstruct.yml; the fluorescence names and an older `BF - Oblique` are per-dataset edits. The reconstruct-with-nextflow check script also validates project.yml both ways (a listed channel the plate lacks is a MISS, an assembled channel the config omits is NOT PROJECTED) and track.yml against the projected plate. The skill carries unprojected channels into the pre-run plan for the user to decide, runs the rename on the projected plate too, and suggests the `BF - Oblique` edit in both reconstruct.yml and project.yml when the raw store names it so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(project): float32 plate for any max/mean/sum channel; drop float64 accumulation NumPy already accumulates safely (float64 for integer input, pairwise float32 for float32 input), so mean and sum no longer force a float64 temporary. The output plate is float32 whenever a channel uses max, mean or sum -- float32 in and out is the common case -- and keeps the input dtype only when every channel is a slice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(project): focus record keys are z and z_per_t The focus record under the output position's biahub-project metadata is focus: {channel, per, z, z_per_t}; the focus_ prefix repeated the block name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(project)!: z selection is start/stop or center with below/above `range` meant absolute planes without a center and offsets with one. The two forms now have their own keys: `start`/`stop` are absolute, half-open like concatenate's time_indices; `below`/`above` count planes around `center`, selecting [center - below, center + above), so the A549 window is `center: focus, below: 20, above: 40`. The validator refuses mixing them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(nextflow): cleanup never targets the projected plate The projected store is a deliverable like the assembled plate: cleanup removes only its slurm_output/ and .iohub-progress/, as the wiring already did; say so in the comment. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(project): chunks_czyx and shards_ratio; A549 writes 16-timepoint shards ProjectionSettings takes the output chunk shape and shards ratio with concatenate's names and semantics (shard = chunks * ratio, C not sharded) and passes them to create_empty_plate; None keeps iohub's defaults. The A549 config, following the Dynamic Cell Atlas array standard where a 2D float32 plate can: one-frame 256 x 256 chunks, the grid the assembled and tracking stores share, and shards of 16 timepoints by the full frame (the standard's T-shard floor; ~150 MB each, 30 files per position instead of 400). iohub writes one shard's timepoints per unit, so resume is 16 frames coarse. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf(project): find the focus of the timepoints in a process pool The focus pre-pass ran the timepoints one after another while the job's other workers idled; with per-timepoint focus on a long timelapse it was the serial part of the step. The timepoints are independent, so with more than one worker they now run in a spawned ProcessPoolExecutor sized by the job's num_workers (the count the projection itself uses), each worker reading its own volume. One worker keeps the in-process loop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(nextflow): projected plate lives beside the assembled one as <ds>_proj.zarr The projected plate is the 2D companion of the assembled plate, not a step of its own: the pipeline now writes it to <N>-assemble/<DATASET>_proj.zarr, so project has no step directory and track stays 5-track. Cleanup already covers the shared directory's slurm_output/ and .iohub-progress/; the slurm_output README notes that a directory can hold two steps' stores. The skill docs, run template and config comments follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Preserve focus metadata Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Validate existing output layout before reuse Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Include channel order in settings fingerprint Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(project): size RAM by the channels read iohub reads one timepoint's (C_read, Z, Y, X) block per worker, so the request must grow with the listed channels: the fixed ram_multiplier of 4 gave 4 GB/CPU whether a config read 2 channels or 6, and the A549 config (6 channels, 3.8 GiB per read, ~5-6 GiB peak) would have hit the cgroup. ram_multiplier is now len(read_indices) + 2 (the read, the window copy and the shard's output buffer): 6 GB/CPU for A549, unchanged for 2 channels. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(project): focus record carries the focus settings Follows c8f8e78, which dumps the focus settings (minus axes) into the biahub-project focus record next to z and z_per_t. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(project): more RAM headroom per worker, calibrated on an A549 position One position of 2026_08_11 SEC61B v8 (6 channels, 86 x 1664 x 1193 float32, 5 shard units busy) peaked at 22 GB max RSS / 28 GB over the process tree, ~5-6 GiB per worker against the 6 GB/CPU requested. +4 volumes requests 7 GB/CPU there, a margin for a timelapse that keeps all 8 workers busy. The time request (70 min for a 3.6 min run) is left as is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): ultrack pin follows royerlab/ultrack#290's pyupgrade fix Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(track)!: cellpose is the default segmentation method TrackingSettings.segmentation_method defaulted to foreground_contour, the legacy path no production config has used since April; a config that omitted the key fell back to it silently. The default is now cellpose, and cellpose without a cellpose_config is refused at load time instead of falling back. The unit tests, which need no weights or GPU, name foreground_contour explicitly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(segment,track)!: segmentation moves out of track into the plate The pipeline becomes assemble -> project -> segment -> track: `biahub segment` writes one OME-NGFF label image per model into the plate it reads (`labels/<model>`, no -o; -o keeps the separate label plate) through iohub's `process_single_position(output_label=...)` (Biohub/iohub#489), so a preempted position resumes. `biahub track` takes that label image as its input (`labels: <model>`, `labels_sigma`) and writes the tracked labels beside it as `labels/<model>_tracked`, with tracks.geff and the CSV. The cellpose code, `cellpose_config` and `target_channel` leave track (both keys are rejected with a pointer to the segment config); `foreground_contour` stays as the legacy path and keeps `target_channel` and `input_images`. Blank frames listed in the CSV are filled with the nearest frame's labels, as the images were before. Track no longer needs a GPU: the `cpu` partition (a new `cpu_reserved` Nextflow label; ultrack cannot resume). mantis-v2.nf gains --segment_config between project and track; segment's init creates the label images so track's init can check that its input exists, and qc_project.yaml counts `labels/nuclei_tracked`. The A549 segment.yml takes the cellpose values tracking was validated with (cpsam_v2, diameter 80, min_size 2500). Label chunking helpers move to biahub.utils.ngff. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(segment,track): scaffold labels with iohub's create_empty_labels; track on preempted The label images are created by iohub's new `create_empty_labels` (Biohub/iohub#489), which takes each label's geometry from its position and records the provenance on the label group, so biahub's wrapper around create_empty_plate goes; `label_layout` keeps only the DCA chunk sizing. Track goes back to the `preempted` partition (the Nextflow `cpu` label): ultrack cannot resume, but with segmentation out of track a position tracks in a minute or two, so a preemption costs that much and the retry covers it. The `cpu_reserved` label is dropped again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(segment,track)!: segment writes only into the plate; track's output_label `biahub segment` loses `-o` and the separate label plate: its labels are the plate's label images, one per model, and segmentation strategies accumulate there as differently named models. A name is only rewritten by the model that made it: `--init` refuses a label image whose recorded `biahub-segment` settings (or shape) differ from the current model's, pointing at a new model name. `output_ome_zarr_version` is rejected, since the labels take the plate's version. `biahub track` gains an optional `output_label`, another name for the tracked label image, for several tracking runs of one segmentation side by side; then the GEFF and CSV are `tracks_<output_label>.geff` / `tracks_<output_label>_<fov>.csv`, while the default keeps `tracks.geff` / `tracks_<fov>.csv`, the names the downstream readers expect. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(track)!: the tracked label image is labels/tracks_<labels> The default name of the tracked label image gains the `tracks_` prefix instead of a `_tracked` suffix, so `labels/tracks_nuclei` sits beside `tracks.geff` and `tracks_<fov>.csv`, and an explicit `output_label` names all three the same way (`labels/<output_label>`, `tracks_<output_label>.geff`, `tracks_<output_label>_<fov>.csv`). The A549 QC config counts `tracks_nuclei`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(project): double a retried task's memory after an OOM kill run_project asked for its init estimate verbatim on every retry, so a low estimate failed all attempts with the same 137. Use retry_memory like the other SLURM steps (fix/nextflow-retry-memory-cap carries the 244 GB cap). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(project): max keeps the input dtype; only mean/sum make the plate float32 max picks an existing voxel, so promoting a max-only plate to float32 doubled its storage for nothing and would truncate uint32 labels above 2^24. A store has one dtype, so a plate with any mean/sum channel is still float32 throughout and its max/slice channels are widened on write. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): pin iohub to main at the #491 merge (80b20a6) One pin for the whole chain: main now carries #489 (output_label, create_empty_labels), #488, #461 and #491, so the 16cdc4f branch pin and the dev/pyramid-shards combination branch retire. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(segment,track)!: label images are (T, 1, Z, Y, X) A label image now has the image's axes with one channel rather than TZYX: the DCA array standard wants every array 5D, and OME-NGFF's own layout note reads a label's dimensions against the image's. iohub creates and writes them that way (czbiohub-sf/iohub, label channel axis); here the DCA label chunks become (16, 1, 1, 128, 128), segment's shape check and track's label reads and writes index the channel axis, and the tests and skill notes follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): pin iohub to Biohub/iohub#496 (labels are (T, 1, Z, Y, X)) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): iohub pin follows Biohub/iohub#496's review fixes (9f21979) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): iohub pin follows Biohub/iohub#496 (eee7567) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): pin iohub to main at the #496 merge (0cb3e35) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(labels): size label shards with iohub's 200 MB target `label_layout` passes iohub `label_shards="200MB"` (Biohub/iohub#461: the largest chunk-aligned shard under the target, filled X, Y, Z then T, C never grown) instead of a computed `label_shards_ratio`, which iohub has deprecated. On the A549 projected plate the result is the same geometry as before, one 16-timepoint chunk by the full frame; #430's segment labels use the same constant, so the two branches agree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): the iohub override follows the pin (0cb3e35) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

…empt did not finish (#434) * feat(track): write labels and tracks.geff into the assembled plate Tracking no longer creates a separate 5-track/<ds>.zarr plate holding the labels as an image channel. It writes into each -i position of the assembled plate (#360, #361): - labels/<target_channel> as an OME-NGFF 0.5 label image: uint32, TZYX with the image's full Z, chunks (16,1,128,128) (1 MB, the DCA SHOULD), shards of 16 timepoints by the YX plane. 2D masks go in the middle plane of the tracked z-window (middle of the volume for a full-stack projection, the plane itself for a one-plane window, middle of a projected section); 3D masks fill the window. The track settings and z placement are recorded under `biahub-track` on the label image. - tracks.geff via ultrack's Tracker.to_geff (zarr v3, seg_id = track_id, related_objects pointing at the labels, axes with units and scale). - tracks_<row>_<col>_<fov>.csv, kept for downstream readers for now. `track --init` creates the empty label images with create_empty_plate(label_names=...). -o is now a working directory for the Ultrack databases and SLURM logs. TrackingSettings drops output_ome_zarr_version, since the labels take the input store's version. Nextflow: --qc_track_config and qc_track.yaml are removed. a549/qc.yaml counts instances in the label image (imaging-qc#243, `label:`), so QC is planned after track --init and computed after tracking. Pins: iohub dev/dca-labels (iohub#485, #486), the ieivanov/ultrack fork (royerlab/ultrack#288), imaging-qc f5ae656, geff/geff-spec >= 1.3. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): repin iohub dev/dca-labels after #485 drops its legacy labels field Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): repin iohub dev/dca-labels after iohub#486 label-default fix Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(track): keep no Ultrack databases by default Each worker now builds its Ultrack database in a fresh scratch directory under its own $TMPDIR and deletes it once the FOV is exported. `-o` becomes optional and keeps the databases in {-o}/{row}_{col}_{fov} when given (e.g. to re-solve with other weights); the SLURM logs go to {-o}/slurm_output, else ./slurm_output, since submitit's folder must be on a shared filesystem. The scratch directory also holds ultrack's own temporary zarr arrays: create_zarr backs its default store with a TemporaryDirectory it lets go out of scope, so those directories were recreated by the first write and left in $TMPDIR, which on Bruno outlives the job. The full resolved Ultrack config (defaults included) is recorded under `biahub-track.ultrack_config` on each label image, so provenance no longer depends on config.toml beside the database. The worker returns nothing rather than a label array inside the deleted scratch directory. Nextflow passes no -o, so tracking has no step directory (no 5-track/, no README, no chmod of the SQLite file) and takes no number; its real logs stay in nextflow/slurm_output/track/. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): pin iohub to iohub#486 now that #485 is on main Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): repin iohub#486 after per-position label defaults fix Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): repin iohub#486 after review fixes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): pin ultrack to royerlab main now that #288 and #289 merged Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * refactor(track): drop the tempfile redirect now that ultrack#289 merged create_zarr now deletes its temporary store with the array, so the per-FOV temporary directory only needs to hold the Ultrack database again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * build(deps): pin iohub to main now that #486 merged Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(project)!: biahub project CLI; segment and track no longer project Add `biahub project`, which turns a 3D plate into a 2D (Z=1) plate with one projection per output channel: max, mean, sum over a Z range or a window around a plane, or one plane (`slice`). The plane may be fixed or the in-focus plane found with the registration engine's focus criterion (FocusSettings subclass: centre crop, NA, wavelength), once per position (median over T) or per timepoint. Ranges are half-open like concatenate's Z_slice and relative to `center` when one is set, so the A549 60-plane window is `center: focus, range: [-20, 40]`. Unlisted input channels are dropped with a warning. The focus plane of every timepoint is recorded in the output position's `biahub-project` metadata. Segment and track stop selecting and projecting Z-planes. Segment loses `z_slice_2D`: a model without do_3D segments in 2D and needs a Z=1 input. Track loses `z_slicing` and the focus window; `output_mode` is checked against the input Z. Both raise, pointing at `biahub project`, when a 3D input meets a 2D request. Old keys are rejected with a migration message. Nextflow: a project step between assemble and track (`5-project`, `6-track` for A549); track reads the projected plate when project runs. The A549 project config reproduces the previous tracking input (mean over the 60-plane focus window on Phase3D), measured on the 800x800 centre crop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(nextflow): A549 project.yml projects every assembled channel; skill checks it The A549 projection config lists every channel of a typical assembled plate under the pre-rename names project sees inside the pipeline: the in-focus plane of BF and Phase3D, the full-stack max of the raw fluorescence channels, and the mean of the virtual-stain predictions over the 60-plane focus window. The focus is found on Phase3D per timepoint, as biahub#400 asks for inference. The brightfield key is `BF`, matching reconstruct.yml; the fluorescence names and an older `BF - Oblique` are per-dataset edits. The reconstruct-with-nextflow check script also validates project.yml both ways (a listed channel the plate lacks is a MISS, an assembled channel the config omits is NOT PROJECTED) and track.yml against the projected plate. The skill carries unprojected channels into the pre-run plan for the user to decide, runs the rename on the projected plate too, and suggests the `BF - Oblique` edit in both reconstruct.yml and project.yml when the raw store names it so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(project): float32 plate for any max/mean/sum channel; drop float64 accumulation NumPy already accumulates safely (float64 for integer input, pairwise float32 for float32 input), so mean and sum no longer force a float64 temporary. The output plate is float32 whenever a channel uses max, mean or sum -- float32 in and out is the common case -- and keeps the input dtype only when every channel is a slice. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(project): focus record keys are z and z_per_t The focus record under the output position's biahub-project metadata is focus: {channel, per, z, z_per_t}; the focus_ prefix repeated the block name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(project)!: z selection is start/stop or center with below/above `range` meant absolute planes without a center and offsets with one. The two forms now have their own keys: `start`/`stop` are absolute, half-open like concatenate's time_indices; `below`/`above` count planes around `center`, selecting [center - below, center + above), so the A549 window is `center: focus, below: 20, above: 40`. The validator refuses mixing them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(nextflow): cleanup never targets the projected plate The projected store is a deliverable like the assembled plate: cleanup removes only its slurm_output/ and .iohub-progress/, as the wiring already did; say so in the comment. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(project): chunks_czyx and shards_ratio; A549 writes 16-timepoint shards ProjectionSettings takes the output chunk shape and shards ratio with concatenate's names and semantics (shard = chunks * ratio, C not sharded) and passes them to create_empty_plate; None keeps iohub's defaults. The A549 config, following the Dynamic Cell Atlas array standard where a 2D float32 plate can: one-frame 256 x 256 chunks, the grid the assembled and tracking stores share, and shards of 16 timepoints by the full frame (the standard's T-shard floor; ~150 MB each, 30 files per position instead of 400). iohub writes one shard's timepoints per unit, so resume is 16 frames coarse. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * perf(project): find the focus of the timepoints in a process pool The focus pre-pass ran the timepoints one after another while the job's other workers idled; with per-timepoint focus on a long timelapse it was the serial part of the step. The timepoints are independent, so with more than one worker they now run in a spawned ProcessPoolExecutor sized by the job's num_workers (the count the projection itself uses), each worker reading its own volume. One worker keeps the in-process loop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(nextflow): projected plate lives beside the assembled one as <ds>_proj.zarr The projected plate is the 2D companion of the assembled plate, not a step of its own: the pipeline now writes it to <N>-assemble/<DATASET>_proj.zarr, so project has no step directory and track stays 5-track. Cleanup already covers the shared directory's slurm_output/ and .iohub-progress/; the slurm_output README notes that a directory can hold two steps' stores. The skill docs, run template and config comments follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Preserve focus metadata Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Validate existing output layout before reuse Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Include channel order in settings fingerprint Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(project): size RAM by the channels read iohub reads one timepoint's (C_read, Z, Y, X) block per worker, so the request must grow with the listed channels: the fixed ram_multiplier of 4 gave 4 GB/CPU whether a config read 2 channels or 6, and the A549 config (6 channels, 3.8 GiB per read, ~5-6 GiB peak) would have hit the cgroup. ram_multiplier is now len(read_indices) + 2 (the read, the window copy and the shard's output buffer): 6 GB/CPU for A549, unchanged for 2 channels. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(project): focus record carries the focus settings Follows c8f8e78, which dumps the focus settings (minus axes) into the biahub-project focus record next to z and z_per_t. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(project): more RAM headroom per worker, calibrated on an A549 position One position of 2026_08_11 SEC61B v8 (6 channels, 86 x 1664 x 1193 float32, 5 shard units busy) peaked at 22 GB max RSS / 28 GB over the process tree, ~5-6 GiB per worker against the 6 GB/CPU requested. +4 volumes requests 7 GB/CPU there, a margin for a timelapse that keeps all 8 workers busy. The time request (70 min for a 3.6 min run) is left as is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): ultrack pin follows royerlab/ultrack#290's pyupgrade fix Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(track)!: cellpose is the default segmentation method TrackingSettings.segmentation_method defaulted to foreground_contour, the legacy path no production config has used since April; a config that omitted the key fell back to it silently. The default is now cellpose, and cellpose without a cellpose_config is refused at load time instead of falling back. The unit tests, which need no weights or GPU, name foreground_contour explicitly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(pyramid): pyramids on images and label images; mantis-v2 adds them last (#359) `biahub pyramid` is rewritten to the processing-command template and computes pyramids on each position's image and on every label image under labels/, in place: --levels (default 3, the DCA v0.2 target), --method for the images and --label-method (max/stride/mode, default max) for the labels, so no level holds a label id level 0 does not. Spatial axes of size 1 are not downsampled, so the projected Z=1 plate gets factor 2 in Y and X only and keeps its Z scale. --cluster/--monitor/--init replace --local; --init refuses a plate carrying a pyramid of another depth and prints RESOURCES. PyramidSettings validates the options and is the biahub-pyramid provenance record on each position. mantis-v2 gains --pyramid_levels (default 3, 1 = off) and a pyramid module run once per plate, assembled and projected, as the LAST writer of each: after tracking and after QC's compute, since all three rewrite the position's zarr.json. Level 0 is never touched. iohub is pinned to Biohub/iohub#488 (PositionLabel.compute_pyramid via a shared MultiscaleNode; mode downsampler fix). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(nextflow): QC is the last step; pyramids run after tracking and ahead of it QC's compute is gated on each plate's pyramid (or on tracking / the step that created the plate when pyramids are off) rather than running concurrently with projection and tracking, so imaging-qc's position-attr stamps never race another writer of the same zarr.json and the report describes the finished plates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): iohub pin follows the #488 review fixes (label-safe defaults, pyramid_dims) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): iohub pin follows the second #488 review round Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): iohub pin follows the third #488 review round Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(segment,track)!: segmentation moves out of track into the plate The pipeline becomes assemble -> project -> segment -> track: `biahub segment` writes one OME-NGFF label image per model into the plate it reads (`labels/<model>`, no -o; -o keeps the separate label plate) through iohub's `process_single_position(output_label=...)` (Biohub/iohub#489), so a preempted position resumes. `biahub track` takes that label image as its input (`labels: <model>`, `labels_sigma`) and writes the tracked labels beside it as `labels/<model>_tracked`, with tracks.geff and the CSV. The cellpose code, `cellpose_config` and `target_channel` leave track (both keys are rejected with a pointer to the segment config); `foreground_contour` stays as the legacy path and keeps `target_channel` and `input_images`. Blank frames listed in the CSV are filled with the nearest frame's labels, as the images were before. Track no longer needs a GPU: the `cpu` partition (a new `cpu_reserved` Nextflow label; ultrack cannot resume). mantis-v2.nf gains --segment_config between project and track; segment's init creates the label images so track's init can check that its input exists, and qc_project.yaml counts `labels/nuclei_tracked`. The A549 segment.yml takes the cellpose values tracking was validated with (cpsam_v2, diameter 80, min_size 2500). Label chunking helpers move to biahub.utils.ngff. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(segment,track): scaffold labels with iohub's create_empty_labels; track on preempted The label images are created by iohub's new `create_empty_labels` (Biohub/iohub#489), which takes each label's geometry from its position and records the provenance on the label group, so biahub's wrapper around create_empty_plate goes; `label_layout` keeps only the DCA chunk sizing. Track goes back to the `preempted` partition (the Nextflow `cpu` label): ultrack cannot resume, but with segmentation out of track a position tracks in a minute or two, so a preemption costs that much and the retry covers it. The `cpu_reserved` label is dropped again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): iohub pin moves to the #461 branch (pyramid levels keep level-0 shard bytes) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(settings)!: shards replaces shards_ratio in the concatenate and project configs `shards` takes iohub's forms (Biohub/iohub#461): a size target such as "5GB", "XY"/"XYZ", or an explicit TCZYX shape in array elements, so a config no longer carries chunk arithmetic that must be redone per output shape. A config that still says shards_ratio is refused with a message naming the new key. Concatenate reads the shard's T extent back from the plate its init created to size its workers. Track scaffolds its label images with create_empty_plate(label_shards=...), an explicit (16, 1, Y, X) shape capped at 2048 px, the same geometry as before. Templates: a549 and zebrafish concatenate `shards: 5GB` (A549 unchanged at 5 timepoints by the full volume; dynatrack goes from 10 to 9 timepoints, under the DCA 5 GB SHOULD); a549 project `shards: 200MB` (21 frames by the full frame instead of 16). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(track): the tracked label image takes the source label's dtype as well label_layout(source_label=...) copies dtype, chunks and shards; track writes its ids in that dtype and refuses, rather than wraps, ids it cannot hold. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(segment,track)!: segment writes only into the plate; track's output_label `biahub segment` loses `-o` and the separate label plate: its labels are the plate's label images, one per model, and segmentation strategies accumulate there as differently named models. A name is only rewritten by the model that made it: `--init` refuses a label image whose recorded `biahub-segment` settings (or shape) differ from the current model's, pointing at a new model name. `output_ome_zarr_version` is rejected, since the labels take the plate's version. `biahub track` gains an optional `output_label`, another name for the tracked label image, for several tracking runs of one segmentation side by side; then the GEFF and CSV are `tracks_<output_label>.geff` / `tracks_<output_label>_<fov>.csv`, while the default keeps `tracks.geff` / `tracks_<fov>.csv`, the names the downstream readers expect. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(configs): concatenate chunks 1 x 8 x 256 x 256; iohub pin keeps level-0 chunks down the pyramid The templates' float32 chunks go from 4 MiB to 2 MiB (Z 16 -> 8); with shards: 5GB the A549 plate's shards become 6 timepoints by the full volume (4.5 GiB). iohub dev/pyramid-shards now keeps the level-0 chunk shape at every pyramid level, clamped to the level, instead of halving it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(track)!: the tracked label image is labels/tracks_<labels> The default name of the tracked label image gains the `tracks_` prefix instead of a `_tracked` suffix, so `labels/tracks_nuclei` sits beside `tracks.geff` and `tracks_<fov>.csv`, and an explicit `output_label` names all three the same way (`labels/<output_label>`, `tracks_<output_label>.geff`, `tracks_<output_label>_<fov>.csv`). The A549 QC config counts `tracks_nuclei`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(pyramid): --resume recomputes only the shards an interrupted attempt did not finish Each shard of each pyramid level is a resumable unit in iohub (czbiohub-sf/iohub feat/pyramid-resume, on #461): its files are cleared before the write and a marker beside the plate records it once synced. The token is the PyramidSettings fingerprint, so other levels or methods recompute. The Nextflow module passes --resume unconditionally, like every resumable step's. The iohub pin moves once that branch is on GitHub. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(pyramid): size the task from the level-0 shard, measured on an A549 position iohub writes a pyramid level one target shard at a time and reads the source region feeding it, so RAM follows the shard, not the volume: one assembled A549 position (4.1 GiB level-0 shards, levels sized to the same bytes) peaked at 44 GB and took 41.5 min against the previous 16 GB / 30 min request. The estimate now asks 12x the shard bytes (52 GB there, the 16 GB floor on the projected plate) and 0.15 min per (t, c) volume (70 min). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): iohub pin moves to dev/pyramid-shards with the resume branch merged Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): name iohub#491 in the pin comment Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(pyramid): double a retried task's memory after an OOM kill run_pyramid requested the init estimate verbatim on every attempt, so an estimate that was too low failed all five attempts of every position with the same 137 and aborted the run. The v9 debug run of 2026_08_11_A549_SEC61B_DENV asked 16 GB 25 times and was OOM-killed 25 times. Use retry_memory like the QC and virtual-stain processes: the next attempt doubles after a 137 and keeps the same request after a preemption. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(project): double a retried task's memory after an OOM kill run_project asked for its init estimate verbatim on every retry, so a low estimate failed all attempts with the same 137. Use retry_memory like the other SLURM steps (fix/nextflow-retry-memory-cap carries the 244 GB cap). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(pyramid): request 14x the shard and 0.3 min per volume, from v9's 54 positions The v9 debug run of 2026_08_11_A549_SEC61B_DENV measured every position of the assembled plate (level-0 shards 4.51 GiB): 46.5 GB peak RSS each (10.3x the shard) and up to 67.3 min, the slowest on the 120 GB cpu-c nodes. The old 12x / 0.15 asked 56 GB / 70 min, 4% above the slowest position. 14x the shard and 0.3 min per volume ask 64 GB / 130 min there; the projected plate keeps the 16 GB floor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(project): max keeps the input dtype; only mean/sum make the plate float32 max picks an existing voxel, so promoting a max-only plate to float32 doubled its storage for nothing and would truncate uint32 labels above 2^24. A store has one dtype, so a plate with any mean/sum channel is still float32 throughout and its max/slice channels are widened on write. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): pin iohub to main at the #491 merge (80b20a6) One pin for the whole chain: main now carries #489 (output_label, create_empty_labels), #488, #461 and #491, so the 16cdc4f branch pin and the dev/pyramid-shards combination branch retire. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(segment,track)!: label images are (T, 1, Z, Y, X) A label image now has the image's axes with one channel rather than TZYX: the DCA array standard wants every array 5D, and OME-NGFF's own layout note reads a label's dimensions against the image's. iohub creates and writes them that way (czbiohub-sf/iohub, label channel axis); here the DCA label chunks become (16, 1, 1, 128, 128), segment's shape check and track's label reads and writes index the channel axis, and the tests and skill notes follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): pin iohub to Biohub/iohub#496 (labels are (T, 1, Z, Y, X)) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): iohub pin follows Biohub/iohub#496's review fixes (9f21979) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): iohub pin follows Biohub/iohub#496 (eee7567) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): pin iohub to main at the #496 merge (0cb3e35) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(labels): size label shards with iohub's 200 MB target `label_layout` passes iohub `label_shards="200MB"` (Biohub/iohub#461: the largest chunk-aligned shard under the target, filled X, Y, Z then T, C never grown) instead of a computed `label_shards_ratio`, which iohub has deprecated. On the A549 projected plate the result is the same geometry as before, one 16-timepoint chunk by the full frame; #430's segment labels use the same constant, so the two branches agree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * build(deps): the iohub override follows the pin (0cb3e35) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * style: ruff format Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

main moved segmentation out of track (#413: `biahub segment` writes labels/<model> into the plate; track reads `labels:` and writes labels/tracks_<labels>, tracks.geff and the CSV into it) and Z projection into `biahub project` (#424), and added pyramids (#430). The trackers of this PR are reworked on top of that: - tracksdata, hoct and trackastra link the instance labels of `labels:` (the default segmentation_method: labels) instead of running cellpose in track; foreground_contour and output_mode 3D are refused for them. - hoct and trackastra read the image the labels were segmented from: the first channel `biahub segment` recorded in the label image's biahub-segment provenance, or hoct_config/trackastra_config.image_channel when set (override, or for a label image segment did not write). --init checks it exists. - Their outputs follow #413: labels/<tracked_label> (5D) and the CSV. They write no tracks.geff yet (ultrack's export reads its own database; #441) and remove a stale one from an earlier run. - hoct and trackastra request a GPU when biahub submits them; ultrack and tracksdata stay on the CPU partition. - pyproject: main's track extra plus tracksdata (git), hoct and trackastra[ilp], with the gurobipy fork by Python version; uv.lock regenerated from main's. - Tests and the track-with-biahub skill moved to labels written into the plate (project -> segment -> one track run per tracker, side by side by output_label). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

ieivanov and others added 30 commits October 7, 2026 18:25

build(deps): repin iohub dev/dca-labels after #485 drops its legacy l…

d8e06a0

…abels field Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

build(deps): repin iohub dev/dca-labels after iohub#486 label-default…

02e5f63

… fix Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

build(deps): pin iohub to iohub#486 now that #485 is on main

662e53f

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

build(deps): repin iohub#486 after per-position label defaults fix

8d473e8

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

build(deps): repin iohub#486 after review fixes

0c35375

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

build(deps): pin ultrack to royerlab main now that #288 and #289 merged

cee19fb

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

refactor(track): drop the tempfile redirect now that ultrack#289 merged

324ff4d

create_zarr now deletes its temporary store with the array, so the per-FOV temporary directory only needs to hold the Ultrack database again. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

build(deps): pin iohub to main now that #486 merged

cbeedd8

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

refactor(project): focus record keys are z and z_per_t

1b240c2

The focus record under the output position's biahub-project metadata is focus: {channel, per, z, z_per_t}; the focus_ prefix repeated the block name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Preserve focus metadata

c8f8e78

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Validate existing output layout before reuse

f06223b

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Include channel order in settings fingerprint

6e5b5e4

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

test(project): focus record carries the focus settings

1bf0071

Follows c8f8e78, which dumps the focus settings (minus axes) into the biahub-project focus record next to z and z_per_t. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

build(deps): ultrack pin follows royerlab/ultrack#290's pyupgrade fix

af5ec1f

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into feat/project-cli

a4f5398

Merge branch 'feat/project-cli' into feat/track-labels-geff

a4f4efa

ieivanov and others added 2 commits October 22, 2026 11:40

ieivanov mentioned this pull request Oct 22, 2026

fix(nextflow): every SLURM step grows memory and time on retry; cap memory at 244 GB #436

Merged

aofei-liu approved these changes Oct 22, 2026

View reviewed changes

ieivanov and others added 17 commits October 23, 2026 13:43

Merge remote-tracking branch 'origin/main' into feat/project-cli

3b689f9

Merge commit '3b689f99' into feat/track-labels-geff

fafdec8

Merge branch 'feat/track-labels-geff' into feat/pyramid-labels

67e7156

Merge remote-tracking branch 'origin/main' into feat/track-labels-geff

2214e79

Merge branch 'feat/track-labels-geff' into feat/pyramid-labels

1be37b0

build(deps): pin iohub to Biohub/iohub#496 (labels are (T, 1, Z, Y, X))

ee6f1a1

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

build(deps): iohub pin follows Biohub/iohub#496's review fixes (9f21979)

ca5fdee

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

build(deps): iohub pin follows Biohub/iohub#496 (eee7567)

fe09c0d

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

build(deps): pin iohub to main at the #496 merge (0cb3e35)

d570380

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

build(deps): the iohub override follows the pin (0cb3e35)

6d9d19a

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge feat/track-labels-geff: label images are (T, 1, Z, Y, X); iohub…

8a92abb

… pin 0cb3e35 label_layout keeps source_label (the tracked labels copy the segment labels' dtype, chunks and shards) on the 5D layout; the pyramid tests create and expect (T, 1, Z, Y, X) label levels.

style: ruff format

d2b0992

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into feat/pyramid-labels

04d6882

ieivanov marked this pull request as ready for review October 24, 2026 00:37

Base automatically changed from feat/track-labels-geff to main October 24, 2026 00:39

ieivanov merged commit dd42cc0 into main Oct 24, 2026
7 checks passed

ieivanov deleted the feat/pyramid-labels branch October 24, 2026 00:42

ieivanov mentioned this pull request Oct 26, 2026

feat(track): choose the tracker: ultrack, tracksdata, HOCT or Trackastra #427

Merged

Navigation Menu

Repository navigation

feat(pyramid): pyramids on images and label images; mantis-v2 adds them after tracking and QC (#359) - #430

feat(pyramid): pyramids on images and label images; mantis-v2 adds them after tracking and QC (#359)#430
ieivanov merged 68 commits into
mainfrom
feat/pyramid-labels

ieivanov commented Oct 21, 2026 •

edited

Loading

Uh oh!

aofei-liu left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ieivanov commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Testing

Follow-ups

Uh oh!

aofei-liu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ieivanov commented Oct 21, 2026 •

edited

Loading

aofei-liu left a comment •

edited

Loading