Skip to content

feat(ngff): write label images through process_single_position; create_empty_labels - #489

Merged
ieivanov merged 3 commits into
mainfrom
feat/process-single-position-output-label
Oct 23, 2026
Merged

ieivanov merged 3 commits into
mainfrom
feat/process-single-position-output-label

Conversation

@ieivanov

@ieivanov ieivanov commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Two additions for a step that segments the plate it reads and writes its labels into that plate's label images (Biohub/biahub#413 moves cellpose segmentation out of biahub track into biahub segment, which does exactly that and needs to resume after a preemption like every other step):

  • process_single_position(..., output_label=<name>) writes the transform's result into the label image labels/<name> of the output position instead of its image. Everything else is unchanged: the same shard-aligned write units, torn-shard repair and --resume markers (kept under the label's name beside the image's), the same extra_metadata on the position. A label image is TZYX, so the transform returns one channel per timepoint (dropped on the way in) and a single input channel group feeds it; output_channel_indices is ignored. The label image must exist; a missing one, several channel groups, or a multi-channel result are refused with a message.
  • create_empty_labels(store_path, position_keys, label_names, ...) is the label-image half of create_empty_plate for a plate that already has its positions: it scaffolds labels/<name> without restating the plate's channels, shape or scale (each label takes its own position's image geometry, as create_empty_plate(label_names=...) already does for existing positions) and records the step's provenance on each label group through label_metadata. The label-creation and layout-validation code is shared with create_empty_plate.

Test plan

  • tests/ngff/test_ngff_utils.py: test_process_single_position_writes_a_label_image_of_the_input (serial and 2 workers; values, image untouched, resume skips, markers under labels/<name>), test_process_single_position_label_output_refusals, test_create_empty_labels_in_existing_positions (0.4 and 0.5; explicit layout, transforms, existing label untouched, metadata on the group, second call a no-op), test_create_empty_labels_refusals.
  • biahub segment --init / biahub track --init on this branch create their label images with create_empty_labels, and biahub segment fills them with --resume (biahub#413's tests, and an end-to-end run on a real plate).
  • Note: while running the full file, hypothesis found a pre-existing failure in test_apply_transform_to_tczyx_and_save that reproduces on main (c9832a7) with the same example (channel chunks of 2, shards_ratio=(2, 1, 1, 2, 2), a slice over all 3 channels → RuntimeError: incompatible offset): apply_transform_to_tczyx_and_save does not refuse a channel-sharded store the way process_single_position does. Not touched here.

🤖 Generated with Claude Code

`output_label=<name>` sends the transform's result into `labels/<name>` of
the output position instead of its image. A label image is TZYX, so the
transform returns one channel per timepoint and a single input channel
group feeds it; the shard-aligned write units, repair and resume markers
work as for the image, with the markers kept under the label's name.
Lets a segmentation step write its labels into the plate it reads (the
label image create_empty_plate's label_names scaffolds) with --resume.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ieivanov added a commit to Biohub/biahub that referenced this pull request Oct 21, 2026
The pipeline becomes assemble -> project -> segment -> track: `biahub segment`
writes one OME-NGFF label image per model into the plate it reads
(`labels/<model>`, no -o; -o keeps the separate label plate) through iohub's
`process_single_position(output_label=...)` (Biohub/iohub#489), so a
preempted position resumes. `biahub track` takes that label image as its
input (`labels: <model>`, `labels_sigma`) and writes the tracked labels beside
it as `labels/<model>_tracked`, with tracks.geff and the CSV. The cellpose
code, `cellpose_config` and `target_channel` leave track (both keys are
rejected with a pointer to the segment config); `foreground_contour` stays as
the legacy path and keeps `target_channel` and `input_images`. Blank frames
listed in the CSV are filled with the nearest frame's labels, as the images
were before. Track no longer needs a GPU: the `cpu` partition (a new
`cpu_reserved` Nextflow label; ultrack cannot resume).

mantis-v2.nf gains --segment_config between project and track; segment's init
creates the label images so track's init can check that its input exists, and
qc_project.yaml counts `labels/nuclei_tracked`. The A549 segment.yml takes the
cellpose values tracking was validated with (cpsam_v2, diameter 80, min_size
2500). Label chunking helpers move to biahub.utils.ngff.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… plate

The label-image half of create_empty_plate for a plate that already has its
positions: a step that segments the plate it reads creates its
`labels/<name>` images without restating the plate's channels, shape or
scale, which each label takes from its own position's image, and records
its provenance on each label group (`label_metadata`). The label-creation
and layout-validation code is shared with create_empty_plate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ieivanov ieivanov changed the title feat(ngff): process_single_position can write into a label image feat(ngff): write label images through process_single_position; create_empty_labels Oct 21, 2026
ieivanov added a commit to Biohub/biahub that referenced this pull request Oct 21, 2026
…bels; track on preempted

The label images are created by iohub's new `create_empty_labels`
(Biohub/iohub#489), which takes each label's geometry from its position
and records the provenance on the label group, so biahub's wrapper around
create_empty_plate goes; `label_layout` keeps only the DCA chunk sizing.

Track goes back to the `preempted` partition (the Nextflow `cpu` label):
ultrack cannot resume, but with segmentation out of track a position tracks
in a minute or two, so a preemption costs that much and the retry covers it.
The `cpu_reserved` label is dropped again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ieivanov
ieivanov merged commit 164c417 into main Oct 23, 2026
19 checks passed
@ieivanov
ieivanov deleted the feat/process-single-position-output-label branch October 23, 2026 22:01
ieivanov added a commit that referenced this pull request Oct 23, 2026
create_empty_plate's label_shards rides along main's _validate_label_layout /
_create_missing_labels refactor from #489; create_empty_labels gains
label_shards in the same forms.
ieivanov added a commit to Biohub/biahub that referenced this pull request Oct 24, 2026
…and tracks.geff (#360, #361) (#413)

* feat(track): write labels and tracks.geff into the assembled plate

Tracking no longer creates a separate 5-track/<ds>.zarr plate holding the
labels as an image channel. It writes into each -i position of the
assembled plate (#360, #361):

- labels/<target_channel> as an OME-NGFF 0.5 label image: uint32, TZYX with
  the image's full Z, chunks (16,1,128,128) (1 MB, the DCA SHOULD), shards
  of 16 timepoints by the YX plane. 2D masks go in the middle plane of the
  tracked z-window (middle of the volume for a full-stack projection, the
  plane itself for a one-plane window, middle of a projected section); 3D
  masks fill the window. The track settings and z placement are recorded
  under `biahub-track` on the label image.
- tracks.geff via ultrack's Tracker.to_geff (zarr v3, seg_id = track_id,
  related_objects pointing at the labels, axes with units and scale).
- tracks_<row>_<col>_<fov>.csv, kept for downstream readers for now.

`track --init` creates the empty label images with
create_empty_plate(label_names=...). -o is now a working directory for the
Ultrack databases and SLURM logs. TrackingSettings drops
output_ome_zarr_version, since the labels take the input store's version.

Nextflow: --qc_track_config and qc_track.yaml are removed. a549/qc.yaml
counts instances in the label image (imaging-qc#243, `label:`), so QC is
planned after track --init and computed after tracking.

Pins: iohub dev/dca-labels (iohub#485, #486), the ieivanov/ultrack fork
(royerlab/ultrack#288), imaging-qc f5ae656, geff/geff-spec >= 1.3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub dev/dca-labels after #485 drops its legacy labels field

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub dev/dca-labels after iohub#486 label-default fix

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(track): keep no Ultrack databases by default

Each worker now builds its Ultrack database in a fresh scratch directory under
its own $TMPDIR and deletes it once the FOV is exported. `-o` becomes optional
and keeps the databases in {-o}/{row}_{col}_{fov} when given (e.g. to re-solve
with other weights); the SLURM logs go to {-o}/slurm_output, else
./slurm_output, since submitit's folder must be on a shared filesystem.

The scratch directory also holds ultrack's own temporary zarr arrays:
create_zarr backs its default store with a TemporaryDirectory it lets go out of
scope, so those directories were recreated by the first write and left in
$TMPDIR, which on Bruno outlives the job.

The full resolved Ultrack config (defaults included) is recorded under
`biahub-track.ultrack_config` on each label image, so provenance no longer
depends on config.toml beside the database. The worker returns nothing rather
than a label array inside the deleted scratch directory.

Nextflow passes no -o, so tracking has no step directory (no 5-track/, no
README, no chmod of the SQLite file) and takes no number; its real logs stay in
nextflow/slurm_output/track/.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): pin iohub to iohub#486 now that #485 is on main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub#486 after per-position label defaults fix

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub#486 after review fixes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): pin ultrack to royerlab main now that #288 and #289 merged

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* refactor(track): drop the tempfile redirect now that ultrack#289 merged

create_zarr now deletes its temporary store with the array, so the per-FOV
temporary directory only needs to hold the Ultrack database again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): pin iohub to main now that #486 merged

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(project)!: biahub project CLI; segment and track no longer project

Add `biahub project`, which turns a 3D plate into a 2D (Z=1) plate with one
projection per output channel: max, mean, sum over a Z range or a window
around a plane, or one plane (`slice`). The plane may be fixed or the in-focus
plane found with the registration engine's focus criterion (FocusSettings
subclass: centre crop, NA, wavelength), once per position (median over T) or
per timepoint. Ranges are half-open like concatenate's Z_slice and relative
to `center` when one is set, so the A549 60-plane window is
`center: focus, range: [-20, 40]`. Unlisted input channels are dropped with a
warning. The focus plane of every timepoint is recorded in the output
position's `biahub-project` metadata.

Segment and track stop selecting and projecting Z-planes. Segment loses
`z_slice_2D`: a model without do_3D segments in 2D and needs a Z=1 input.
Track loses `z_slicing` and the focus window; `output_mode` is checked
against the input Z. Both raise, pointing at `biahub project`, when a 3D
input meets a 2D request. Old keys are rejected with a migration message.

Nextflow: a project step between assemble and track (`5-project`,
`6-track` for A549); track reads the projected plate when project runs.
The A549 project config reproduces the previous tracking input (mean over
the 60-plane focus window on Phase3D), measured on the 800x800 centre crop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(nextflow): A549 project.yml projects every assembled channel; skill checks it

The A549 projection config lists every channel of a typical assembled plate
under the pre-rename names project sees inside the pipeline: the in-focus
plane of BF and Phase3D, the full-stack max of the raw fluorescence channels,
and the mean of the virtual-stain predictions over the 60-plane focus window.
The focus is found on Phase3D per timepoint, as biahub#400 asks for inference.
The brightfield key is `BF`, matching reconstruct.yml; the fluorescence names
and an older `BF - Oblique` are per-dataset edits.

The reconstruct-with-nextflow check script also validates project.yml both
ways (a listed channel the plate lacks is a MISS, an assembled channel the
config omits is NOT PROJECTED) and track.yml against the projected plate. The
skill carries unprojected channels into the pre-run plan for the user to
decide, runs the rename on the projected plate too, and suggests the
`BF - Oblique` edit in both reconstruct.yml and project.yml when the raw
store names it so.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(project): float32 plate for any max/mean/sum channel; drop float64 accumulation

NumPy already accumulates safely (float64 for integer input, pairwise float32
for float32 input), so mean and sum no longer force a float64 temporary. The
output plate is float32 whenever a channel uses max, mean or sum -- float32 in
and out is the common case -- and keeps the input dtype only when every
channel is a slice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(project): focus record keys are z and z_per_t

The focus record under the output position's biahub-project metadata is
focus: {channel, per, z, z_per_t}; the focus_ prefix repeated the block name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(project)!: z selection is start/stop or center with below/above

`range` meant absolute planes without a center and offsets with one. The two
forms now have their own keys: `start`/`stop` are absolute, half-open like
concatenate's time_indices; `below`/`above` count planes around `center`,
selecting [center - below, center + above), so the A549 window is
`center: focus, below: 20, above: 40`. The validator refuses mixing them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(nextflow): cleanup never targets the projected plate

The projected store is a deliverable like the assembled plate: cleanup
removes only its slurm_output/ and .iohub-progress/, as the wiring already
did; say so in the comment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(project): chunks_czyx and shards_ratio; A549 writes 16-timepoint shards

ProjectionSettings takes the output chunk shape and shards ratio with
concatenate's names and semantics (shard = chunks * ratio, C not sharded) and
passes them to create_empty_plate; None keeps iohub's defaults.

The A549 config, following the Dynamic Cell Atlas array standard where a 2D
float32 plate can: one-frame 256 x 256 chunks, the grid the assembled and
tracking stores share, and shards of 16 timepoints by the full frame (the
standard's T-shard floor; ~150 MB each, 30 files per position instead of 400).
iohub writes one shard's timepoints per unit, so resume is 16 frames coarse.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* perf(project): find the focus of the timepoints in a process pool

The focus pre-pass ran the timepoints one after another while the job's
other workers idled; with per-timepoint focus on a long timelapse it was the
serial part of the step. The timepoints are independent, so with more than
one worker they now run in a spawned ProcessPoolExecutor sized by the job's
num_workers (the count the projection itself uses), each worker reading its
own volume. One worker keeps the in-process loop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(nextflow): projected plate lives beside the assembled one as <ds>_proj.zarr

The projected plate is the 2D companion of the assembled plate, not a step
of its own: the pipeline now writes it to <N>-assemble/<DATASET>_proj.zarr,
so project has no step directory and track stays 5-track. Cleanup already
covers the shared directory's slurm_output/ and .iohub-progress/; the
slurm_output README notes that a directory can hold two steps' stores. The
skill docs, run template and config comments follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Preserve focus metadata

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Validate existing output layout before reuse

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Include channel order in settings fingerprint

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix(project): size RAM by the channels read

iohub reads one timepoint's (C_read, Z, Y, X) block per worker, so the
request must grow with the listed channels: the fixed ram_multiplier of 4
gave 4 GB/CPU whether a config read 2 channels or 6, and the A549 config
(6 channels, 3.8 GiB per read, ~5-6 GiB peak) would have hit the cgroup.
ram_multiplier is now len(read_indices) + 2 (the read, the window copy and
the shard's output buffer): 6 GB/CPU for A549, unchanged for 2 channels.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(project): focus record carries the focus settings

Follows c8f8e78, which dumps the focus settings (minus axes) into the
biahub-project focus record next to z and z_per_t.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(project): more RAM headroom per worker, calibrated on an A549 position

One position of 2026_08_11 SEC61B v8 (6 channels, 86 x 1664 x 1193 float32,
5 shard units busy) peaked at 22 GB max RSS / 28 GB over the process tree,
~5-6 GiB per worker against the 6 GB/CPU requested. +4 volumes requests
7 GB/CPU there, a margin for a timelapse that keeps all 8 workers busy. The
time request (70 min for a 3.6 min run) is left as is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): ultrack pin follows royerlab/ultrack#290's pyupgrade fix

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(track)!: cellpose is the default segmentation method

TrackingSettings.segmentation_method defaulted to foreground_contour, the
legacy path no production config has used since April; a config that
omitted the key fell back to it silently. The default is now cellpose, and
cellpose without a cellpose_config is refused at load time instead of
falling back. The unit tests, which need no weights or GPU, name
foreground_contour explicitly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(segment,track)!: segmentation moves out of track into the plate

The pipeline becomes assemble -> project -> segment -> track: `biahub segment`
writes one OME-NGFF label image per model into the plate it reads
(`labels/<model>`, no -o; -o keeps the separate label plate) through iohub's
`process_single_position(output_label=...)` (Biohub/iohub#489), so a
preempted position resumes. `biahub track` takes that label image as its
input (`labels: <model>`, `labels_sigma`) and writes the tracked labels beside
it as `labels/<model>_tracked`, with tracks.geff and the CSV. The cellpose
code, `cellpose_config` and `target_channel` leave track (both keys are
rejected with a pointer to the segment config); `foreground_contour` stays as
the legacy path and keeps `target_channel` and `input_images`. Blank frames
listed in the CSV are filled with the nearest frame's labels, as the images
were before. Track no longer needs a GPU: the `cpu` partition (a new
`cpu_reserved` Nextflow label; ultrack cannot resume).

mantis-v2.nf gains --segment_config between project and track; segment's init
creates the label images so track's init can check that its input exists, and
qc_project.yaml counts `labels/nuclei_tracked`. The A549 segment.yml takes the
cellpose values tracking was validated with (cpsam_v2, diameter 80, min_size
2500). Label chunking helpers move to biahub.utils.ngff.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(segment,track): scaffold labels with iohub's create_empty_labels; track on preempted

The label images are created by iohub's new `create_empty_labels`
(Biohub/iohub#489), which takes each label's geometry from its position
and records the provenance on the label group, so biahub's wrapper around
create_empty_plate goes; `label_layout` keeps only the DCA chunk sizing.

Track goes back to the `preempted` partition (the Nextflow `cpu` label):
ultrack cannot resume, but with segmentation out of track a position tracks
in a minute or two, so a preemption costs that much and the retry covers it.
The `cpu_reserved` label is dropped again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(segment,track)!: segment writes only into the plate; track's output_label

`biahub segment` loses `-o` and the separate label plate: its labels are the
plate's label images, one per model, and segmentation strategies accumulate
there as differently named models. A name is only rewritten by the model that
made it: `--init` refuses a label image whose recorded `biahub-segment`
settings (or shape) differ from the current model's, pointing at a new model
name. `output_ome_zarr_version` is rejected, since the labels take the
plate's version.

`biahub track` gains an optional `output_label`, another name for the tracked
label image, for several tracking runs of one segmentation side by side; then
the GEFF and CSV are `tracks_<output_label>.geff` / `tracks_<output_label>_<fov>.csv`,
while the default keeps `tracks.geff` / `tracks_<fov>.csv`, the names the
downstream readers expect.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(track)!: the tracked label image is labels/tracks_<labels>

The default name of the tracked label image gains the `tracks_` prefix instead
of a `_tracked` suffix, so `labels/tracks_nuclei` sits beside `tracks.geff`
and `tracks_<fov>.csv`, and an explicit `output_label` names all three the
same way (`labels/<output_label>`, `tracks_<output_label>.geff`,
`tracks_<output_label>_<fov>.csv`). The A549 QC config counts
`tracks_nuclei`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(project): double a retried task's memory after an OOM kill

run_project asked for its init estimate verbatim on every retry, so a low
estimate failed all attempts with the same 137. Use retry_memory like the other
SLURM steps (fix/nextflow-retry-memory-cap carries the 244 GB cap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(project): max keeps the input dtype; only mean/sum make the plate float32

max picks an existing voxel, so promoting a max-only plate to float32
doubled its storage for nothing and would truncate uint32 labels above
2^24. A store has one dtype, so a plate with any mean/sum channel is
still float32 throughout and its max/slice channels are widened on write.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): pin iohub to main at the #491 merge (80b20a6)

One pin for the whole chain: main now carries #489 (output_label,
create_empty_labels), #488, #461 and #491, so the 16cdc4f branch pin and
the dev/pyramid-shards combination branch retire.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(segment,track)!: label images are (T, 1, Z, Y, X)

A label image now has the image's axes with one channel rather than TZYX:
the DCA array standard wants every array 5D, and OME-NGFF's own layout note
reads a label's dimensions against the image's. iohub creates and writes
them that way (czbiohub-sf/iohub, label channel axis); here the DCA label
chunks become (16, 1, 1, 128, 128), segment's shape check and track's label
reads and writes index the channel axis, and the tests and skill notes
follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): pin iohub to Biohub/iohub#496 (labels are (T, 1, Z, Y, X))

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows Biohub/iohub#496's review fixes (9f21979)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows Biohub/iohub#496 (eee7567)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): pin iohub to main at the #496 merge (0cb3e35)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(labels): size label shards with iohub's 200 MB target

`label_layout` passes iohub `label_shards="200MB"` (Biohub/iohub#461:
the largest chunk-aligned shard under the target, filled X, Y, Z then T, C
never grown) instead of a computed `label_shards_ratio`, which iohub has
deprecated. On the A549 projected plate the result is the same geometry as
before, one 16-timepoint chunk by the full frame; #430's segment labels use
the same constant, so the two branches agree.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): the iohub override follows the pin (0cb3e35)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
ieivanov added a commit to Biohub/biahub that referenced this pull request Oct 24, 2026
…em after tracking and QC (#359) (#430)

* feat(track): write labels and tracks.geff into the assembled plate

Tracking no longer creates a separate 5-track/<ds>.zarr plate holding the
labels as an image channel. It writes into each -i position of the
assembled plate (#360, #361):

- labels/<target_channel> as an OME-NGFF 0.5 label image: uint32, TZYX with
  the image's full Z, chunks (16,1,128,128) (1 MB, the DCA SHOULD), shards
  of 16 timepoints by the YX plane. 2D masks go in the middle plane of the
  tracked z-window (middle of the volume for a full-stack projection, the
  plane itself for a one-plane window, middle of a projected section); 3D
  masks fill the window. The track settings and z placement are recorded
  under `biahub-track` on the label image.
- tracks.geff via ultrack's Tracker.to_geff (zarr v3, seg_id = track_id,
  related_objects pointing at the labels, axes with units and scale).
- tracks_<row>_<col>_<fov>.csv, kept for downstream readers for now.

`track --init` creates the empty label images with
create_empty_plate(label_names=...). -o is now a working directory for the
Ultrack databases and SLURM logs. TrackingSettings drops
output_ome_zarr_version, since the labels take the input store's version.

Nextflow: --qc_track_config and qc_track.yaml are removed. a549/qc.yaml
counts instances in the label image (imaging-qc#243, `label:`), so QC is
planned after track --init and computed after tracking.

Pins: iohub dev/dca-labels (iohub#485, #486), the ieivanov/ultrack fork
(royerlab/ultrack#288), imaging-qc f5ae656, geff/geff-spec >= 1.3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub dev/dca-labels after #485 drops its legacy labels field

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub dev/dca-labels after iohub#486 label-default fix

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(track): keep no Ultrack databases by default

Each worker now builds its Ultrack database in a fresh scratch directory under
its own $TMPDIR and deletes it once the FOV is exported. `-o` becomes optional
and keeps the databases in {-o}/{row}_{col}_{fov} when given (e.g. to re-solve
with other weights); the SLURM logs go to {-o}/slurm_output, else
./slurm_output, since submitit's folder must be on a shared filesystem.

The scratch directory also holds ultrack's own temporary zarr arrays:
create_zarr backs its default store with a TemporaryDirectory it lets go out of
scope, so those directories were recreated by the first write and left in
$TMPDIR, which on Bruno outlives the job.

The full resolved Ultrack config (defaults included) is recorded under
`biahub-track.ultrack_config` on each label image, so provenance no longer
depends on config.toml beside the database. The worker returns nothing rather
than a label array inside the deleted scratch directory.

Nextflow passes no -o, so tracking has no step directory (no 5-track/, no
README, no chmod of the SQLite file) and takes no number; its real logs stay in
nextflow/slurm_output/track/.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): pin iohub to iohub#486 now that #485 is on main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub#486 after per-position label defaults fix

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub#486 after review fixes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): pin ultrack to royerlab main now that #288 and #289 merged

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* refactor(track): drop the tempfile redirect now that ultrack#289 merged

create_zarr now deletes its temporary store with the array, so the per-FOV
temporary directory only needs to hold the Ultrack database again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): pin iohub to main now that #486 merged

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(project)!: biahub project CLI; segment and track no longer project

Add `biahub project`, which turns a 3D plate into a 2D (Z=1) plate with one
projection per output channel: max, mean, sum over a Z range or a window
around a plane, or one plane (`slice`). The plane may be fixed or the in-focus
plane found with the registration engine's focus criterion (FocusSettings
subclass: centre crop, NA, wavelength), once per position (median over T) or
per timepoint. Ranges are half-open like concatenate's Z_slice and relative
to `center` when one is set, so the A549 60-plane window is
`center: focus, range: [-20, 40]`. Unlisted input channels are dropped with a
warning. The focus plane of every timepoint is recorded in the output
position's `biahub-project` metadata.

Segment and track stop selecting and projecting Z-planes. Segment loses
`z_slice_2D`: a model without do_3D segments in 2D and needs a Z=1 input.
Track loses `z_slicing` and the focus window; `output_mode` is checked
against the input Z. Both raise, pointing at `biahub project`, when a 3D
input meets a 2D request. Old keys are rejected with a migration message.

Nextflow: a project step between assemble and track (`5-project`,
`6-track` for A549); track reads the projected plate when project runs.
The A549 project config reproduces the previous tracking input (mean over
the 60-plane focus window on Phase3D), measured on the 800x800 centre crop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(nextflow): A549 project.yml projects every assembled channel; skill checks it

The A549 projection config lists every channel of a typical assembled plate
under the pre-rename names project sees inside the pipeline: the in-focus
plane of BF and Phase3D, the full-stack max of the raw fluorescence channels,
and the mean of the virtual-stain predictions over the 60-plane focus window.
The focus is found on Phase3D per timepoint, as biahub#400 asks for inference.
The brightfield key is `BF`, matching reconstruct.yml; the fluorescence names
and an older `BF - Oblique` are per-dataset edits.

The reconstruct-with-nextflow check script also validates project.yml both
ways (a listed channel the plate lacks is a MISS, an assembled channel the
config omits is NOT PROJECTED) and track.yml against the projected plate. The
skill carries unprojected channels into the pre-run plan for the user to
decide, runs the rename on the projected plate too, and suggests the
`BF - Oblique` edit in both reconstruct.yml and project.yml when the raw
store names it so.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(project): float32 plate for any max/mean/sum channel; drop float64 accumulation

NumPy already accumulates safely (float64 for integer input, pairwise float32
for float32 input), so mean and sum no longer force a float64 temporary. The
output plate is float32 whenever a channel uses max, mean or sum -- float32 in
and out is the common case -- and keeps the input dtype only when every
channel is a slice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(project): focus record keys are z and z_per_t

The focus record under the output position's biahub-project metadata is
focus: {channel, per, z, z_per_t}; the focus_ prefix repeated the block name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(project)!: z selection is start/stop or center with below/above

`range` meant absolute planes without a center and offsets with one. The two
forms now have their own keys: `start`/`stop` are absolute, half-open like
concatenate's time_indices; `below`/`above` count planes around `center`,
selecting [center - below, center + above), so the A549 window is
`center: focus, below: 20, above: 40`. The validator refuses mixing them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(nextflow): cleanup never targets the projected plate

The projected store is a deliverable like the assembled plate: cleanup
removes only its slurm_output/ and .iohub-progress/, as the wiring already
did; say so in the comment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(project): chunks_czyx and shards_ratio; A549 writes 16-timepoint shards

ProjectionSettings takes the output chunk shape and shards ratio with
concatenate's names and semantics (shard = chunks * ratio, C not sharded) and
passes them to create_empty_plate; None keeps iohub's defaults.

The A549 config, following the Dynamic Cell Atlas array standard where a 2D
float32 plate can: one-frame 256 x 256 chunks, the grid the assembled and
tracking stores share, and shards of 16 timepoints by the full frame (the
standard's T-shard floor; ~150 MB each, 30 files per position instead of 400).
iohub writes one shard's timepoints per unit, so resume is 16 frames coarse.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* perf(project): find the focus of the timepoints in a process pool

The focus pre-pass ran the timepoints one after another while the job's
other workers idled; with per-timepoint focus on a long timelapse it was the
serial part of the step. The timepoints are independent, so with more than
one worker they now run in a spawned ProcessPoolExecutor sized by the job's
num_workers (the count the projection itself uses), each worker reading its
own volume. One worker keeps the in-process loop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(nextflow): projected plate lives beside the assembled one as <ds>_proj.zarr

The projected plate is the 2D companion of the assembled plate, not a step
of its own: the pipeline now writes it to <N>-assemble/<DATASET>_proj.zarr,
so project has no step directory and track stays 5-track. Cleanup already
covers the shared directory's slurm_output/ and .iohub-progress/; the
slurm_output README notes that a directory can hold two steps' stores. The
skill docs, run template and config comments follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Preserve focus metadata

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Validate existing output layout before reuse

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Include channel order in settings fingerprint

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix(project): size RAM by the channels read

iohub reads one timepoint's (C_read, Z, Y, X) block per worker, so the
request must grow with the listed channels: the fixed ram_multiplier of 4
gave 4 GB/CPU whether a config read 2 channels or 6, and the A549 config
(6 channels, 3.8 GiB per read, ~5-6 GiB peak) would have hit the cgroup.
ram_multiplier is now len(read_indices) + 2 (the read, the window copy and
the shard's output buffer): 6 GB/CPU for A549, unchanged for 2 channels.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(project): focus record carries the focus settings

Follows c8f8e78, which dumps the focus settings (minus axes) into the
biahub-project focus record next to z and z_per_t.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(project): more RAM headroom per worker, calibrated on an A549 position

One position of 2026_08_11 SEC61B v8 (6 channels, 86 x 1664 x 1193 float32,
5 shard units busy) peaked at 22 GB max RSS / 28 GB over the process tree,
~5-6 GiB per worker against the 6 GB/CPU requested. +4 volumes requests
7 GB/CPU there, a margin for a timelapse that keeps all 8 workers busy. The
time request (70 min for a 3.6 min run) is left as is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): ultrack pin follows royerlab/ultrack#290's pyupgrade fix

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(track)!: cellpose is the default segmentation method

TrackingSettings.segmentation_method defaulted to foreground_contour, the
legacy path no production config has used since April; a config that
omitted the key fell back to it silently. The default is now cellpose, and
cellpose without a cellpose_config is refused at load time instead of
falling back. The unit tests, which need no weights or GPU, name
foreground_contour explicitly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(pyramid): pyramids on images and label images; mantis-v2 adds them last (#359)

`biahub pyramid` is rewritten to the processing-command template and
computes pyramids on each position's image and on every label image under
labels/, in place: --levels (default 3, the DCA v0.2 target), --method for
the images and --label-method (max/stride/mode, default max) for the labels,
so no level holds a label id level 0 does not. Spatial axes of size 1 are
not downsampled, so the projected Z=1 plate gets factor 2 in Y and X only
and keeps its Z scale. --cluster/--monitor/--init replace --local; --init
refuses a plate carrying a pyramid of another depth and prints RESOURCES.
PyramidSettings validates the options and is the biahub-pyramid provenance
record on each position.

mantis-v2 gains --pyramid_levels (default 3, 1 = off) and a pyramid module
run once per plate, assembled and projected, as the LAST writer of each:
after tracking and after QC's compute, since all three rewrite the
position's zarr.json. Level 0 is never touched.

iohub is pinned to Biohub/iohub#488 (PositionLabel.compute_pyramid via
a shared MultiscaleNode; mode downsampler fix).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(nextflow): QC is the last step; pyramids run after tracking and ahead of it

QC's compute is gated on each plate's pyramid (or on tracking / the step that
created the plate when pyramids are off) rather than running concurrently
with projection and tracking, so imaging-qc's position-attr stamps never race
another writer of the same zarr.json and the report describes the finished
plates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows the #488 review fixes (label-safe defaults, pyramid_dims)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows the second #488 review round

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows the third #488 review round

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(segment,track)!: segmentation moves out of track into the plate

The pipeline becomes assemble -> project -> segment -> track: `biahub segment`
writes one OME-NGFF label image per model into the plate it reads
(`labels/<model>`, no -o; -o keeps the separate label plate) through iohub's
`process_single_position(output_label=...)` (Biohub/iohub#489), so a
preempted position resumes. `biahub track` takes that label image as its
input (`labels: <model>`, `labels_sigma`) and writes the tracked labels beside
it as `labels/<model>_tracked`, with tracks.geff and the CSV. The cellpose
code, `cellpose_config` and `target_channel` leave track (both keys are
rejected with a pointer to the segment config); `foreground_contour` stays as
the legacy path and keeps `target_channel` and `input_images`. Blank frames
listed in the CSV are filled with the nearest frame's labels, as the images
were before. Track no longer needs a GPU: the `cpu` partition (a new
`cpu_reserved` Nextflow label; ultrack cannot resume).

mantis-v2.nf gains --segment_config between project and track; segment's init
creates the label images so track's init can check that its input exists, and
qc_project.yaml counts `labels/nuclei_tracked`. The A549 segment.yml takes the
cellpose values tracking was validated with (cpsam_v2, diameter 80, min_size
2500). Label chunking helpers move to biahub.utils.ngff.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(segment,track): scaffold labels with iohub's create_empty_labels; track on preempted

The label images are created by iohub's new `create_empty_labels`
(Biohub/iohub#489), which takes each label's geometry from its position
and records the provenance on the label group, so biahub's wrapper around
create_empty_plate goes; `label_layout` keeps only the DCA chunk sizing.

Track goes back to the `preempted` partition (the Nextflow `cpu` label):
ultrack cannot resume, but with segmentation out of track a position tracks
in a minute or two, so a preemption costs that much and the retry covers it.
The `cpu_reserved` label is dropped again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin moves to the #461 branch (pyramid levels keep level-0 shard bytes)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(settings)!: shards replaces shards_ratio in the concatenate and project configs

`shards` takes iohub's forms (Biohub/iohub#461): a size target such as
"5GB", "XY"/"XYZ", or an explicit TCZYX shape in array elements, so a config
no longer carries chunk arithmetic that must be redone per output shape. A
config that still says shards_ratio is refused with a message naming the new
key. Concatenate reads the shard's T extent back from the plate its init
created to size its workers. Track scaffolds its label images with
create_empty_plate(label_shards=...), an explicit (16, 1, Y, X) shape capped
at 2048 px, the same geometry as before.

Templates: a549 and zebrafish concatenate `shards: 5GB` (A549 unchanged at 5
timepoints by the full volume; dynatrack goes from 10 to 9 timepoints, under
the DCA 5 GB SHOULD); a549 project `shards: 200MB` (21 frames by the full
frame instead of 16).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(track): the tracked label image takes the source label's dtype as well

label_layout(source_label=...) copies dtype, chunks and shards; track writes
its ids in that dtype and refuses, rather than wraps, ids it cannot hold.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(segment,track)!: segment writes only into the plate; track's output_label

`biahub segment` loses `-o` and the separate label plate: its labels are the
plate's label images, one per model, and segmentation strategies accumulate
there as differently named models. A name is only rewritten by the model that
made it: `--init` refuses a label image whose recorded `biahub-segment`
settings (or shape) differ from the current model's, pointing at a new model
name. `output_ome_zarr_version` is rejected, since the labels take the
plate's version.

`biahub track` gains an optional `output_label`, another name for the tracked
label image, for several tracking runs of one segmentation side by side; then
the GEFF and CSV are `tracks_<output_label>.geff` / `tracks_<output_label>_<fov>.csv`,
while the default keeps `tracks.geff` / `tracks_<fov>.csv`, the names the
downstream readers expect.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(configs): concatenate chunks 1 x 8 x 256 x 256; iohub pin keeps level-0 chunks down the pyramid

The templates' float32 chunks go from 4 MiB to 2 MiB (Z 16 -> 8); with
shards: 5GB the A549 plate's shards become 6 timepoints by the full volume
(4.5 GiB). iohub dev/pyramid-shards now keeps the level-0 chunk shape at
every pyramid level, clamped to the level, instead of halving it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(track)!: the tracked label image is labels/tracks_<labels>

The default name of the tracked label image gains the `tracks_` prefix instead
of a `_tracked` suffix, so `labels/tracks_nuclei` sits beside `tracks.geff`
and `tracks_<fov>.csv`, and an explicit `output_label` names all three the
same way (`labels/<output_label>`, `tracks_<output_label>.geff`,
`tracks_<output_label>_<fov>.csv`). The A549 QC config counts
`tracks_nuclei`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(pyramid): size the task from the level-0 shard, measured on an A549 position

iohub writes a pyramid level one target shard at a time and reads the source
region feeding it, so RAM follows the shard, not the volume: one assembled
A549 position (4.1 GiB level-0 shards, levels sized to the same bytes) peaked
at 44 GB and took 41.5 min against the previous 16 GB / 30 min request. The
estimate now asks 12x the shard bytes (52 GB there, the 16 GB floor on the
projected plate) and 0.15 min per (t, c) volume (70 min).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(pyramid): double a retried task's memory after an OOM kill

run_pyramid requested the init estimate verbatim on every attempt, so an
estimate that was too low failed all five attempts of every position with the
same 137 and aborted the run. The v9 debug run of 2026_08_11_A549_SEC61B_DENV
asked 16 GB 25 times and was OOM-killed 25 times. Use retry_memory like the QC
and virtual-stain processes: the next attempt doubles after a 137 and keeps
the same request after a preemption.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(project): double a retried task's memory after an OOM kill

run_project asked for its init estimate verbatim on every retry, so a low
estimate failed all attempts with the same 137. Use retry_memory like the other
SLURM steps (fix/nextflow-retry-memory-cap carries the 244 GB cap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pyramid): request 14x the shard and 0.3 min per volume, from v9's 54 positions

The v9 debug run of 2026_08_11_A549_SEC61B_DENV measured every position of
the assembled plate (level-0 shards 4.51 GiB): 46.5 GB peak RSS each (10.3x
the shard) and up to 67.3 min, the slowest on the 120 GB cpu-c nodes. The old
12x / 0.15 asked 56 GB / 70 min, 4% above the slowest position. 14x the shard
and 0.3 min per volume ask 64 GB / 130 min there; the projected plate keeps
the 16 GB floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(project): max keeps the input dtype; only mean/sum make the plate float32

max picks an existing voxel, so promoting a max-only plate to float32
doubled its storage for nothing and would truncate uint32 labels above
2^24. A store has one dtype, so a plate with any mean/sum channel is
still float32 throughout and its max/slice channels are widened on write.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): pin iohub to main at the #491 merge (80b20a6)

One pin for the whole chain: main now carries #489 (output_label,
create_empty_labels), #488, #461 and #491, so the 16cdc4f branch pin and
the dev/pyramid-shards combination branch retire.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(segment,track)!: label images are (T, 1, Z, Y, X)

A label image now has the image's axes with one channel rather than TZYX:
the DCA array standard wants every array 5D, and OME-NGFF's own layout note
reads a label's dimensions against the image's. iohub creates and writes
them that way (czbiohub-sf/iohub, label channel axis); here the DCA label
chunks become (16, 1, 1, 128, 128), segment's shape check and track's label
reads and writes index the channel axis, and the tests and skill notes
follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): pin iohub to Biohub/iohub#496 (labels are (T, 1, Z, Y, X))

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows Biohub/iohub#496's review fixes (9f21979)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows Biohub/iohub#496 (eee7567)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): pin iohub to main at the #496 merge (0cb3e35)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(labels): size label shards with iohub's 200 MB target

`label_layout` passes iohub `label_shards="200MB"` (Biohub/iohub#461:
the largest chunk-aligned shard under the target, filled X, Y, Z then T, C
never grown) instead of a computed `label_shards_ratio`, which iohub has
deprecated. On the A549 projected plate the result is the same geometry as
before, one 16-timepoint chunk by the full frame; #430's segment labels use
the same constant, so the two branches agree.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): the iohub override follows the pin (0cb3e35)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* style: ruff format

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
ieivanov added a commit to Biohub/biahub that referenced this pull request Oct 24, 2026
…empt did not finish (#434)

* feat(track): write labels and tracks.geff into the assembled plate

Tracking no longer creates a separate 5-track/<ds>.zarr plate holding the
labels as an image channel. It writes into each -i position of the
assembled plate (#360, #361):

- labels/<target_channel> as an OME-NGFF 0.5 label image: uint32, TZYX with
  the image's full Z, chunks (16,1,128,128) (1 MB, the DCA SHOULD), shards
  of 16 timepoints by the YX plane. 2D masks go in the middle plane of the
  tracked z-window (middle of the volume for a full-stack projection, the
  plane itself for a one-plane window, middle of a projected section); 3D
  masks fill the window. The track settings and z placement are recorded
  under `biahub-track` on the label image.
- tracks.geff via ultrack's Tracker.to_geff (zarr v3, seg_id = track_id,
  related_objects pointing at the labels, axes with units and scale).
- tracks_<row>_<col>_<fov>.csv, kept for downstream readers for now.

`track --init` creates the empty label images with
create_empty_plate(label_names=...). -o is now a working directory for the
Ultrack databases and SLURM logs. TrackingSettings drops
output_ome_zarr_version, since the labels take the input store's version.

Nextflow: --qc_track_config and qc_track.yaml are removed. a549/qc.yaml
counts instances in the label image (imaging-qc#243, `label:`), so QC is
planned after track --init and computed after tracking.

Pins: iohub dev/dca-labels (iohub#485, #486), the ieivanov/ultrack fork
(royerlab/ultrack#288), imaging-qc f5ae656, geff/geff-spec >= 1.3.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub dev/dca-labels after #485 drops its legacy labels field

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub dev/dca-labels after iohub#486 label-default fix

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(track): keep no Ultrack databases by default

Each worker now builds its Ultrack database in a fresh scratch directory under
its own $TMPDIR and deletes it once the FOV is exported. `-o` becomes optional
and keeps the databases in {-o}/{row}_{col}_{fov} when given (e.g. to re-solve
with other weights); the SLURM logs go to {-o}/slurm_output, else
./slurm_output, since submitit's folder must be on a shared filesystem.

The scratch directory also holds ultrack's own temporary zarr arrays:
create_zarr backs its default store with a TemporaryDirectory it lets go out of
scope, so those directories were recreated by the first write and left in
$TMPDIR, which on Bruno outlives the job.

The full resolved Ultrack config (defaults included) is recorded under
`biahub-track.ultrack_config` on each label image, so provenance no longer
depends on config.toml beside the database. The worker returns nothing rather
than a label array inside the deleted scratch directory.

Nextflow passes no -o, so tracking has no step directory (no 5-track/, no
README, no chmod of the SQLite file) and takes no number; its real logs stay in
nextflow/slurm_output/track/.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): pin iohub to iohub#486 now that #485 is on main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub#486 after per-position label defaults fix

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): repin iohub#486 after review fixes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): pin ultrack to royerlab main now that #288 and #289 merged

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* refactor(track): drop the tempfile redirect now that ultrack#289 merged

create_zarr now deletes its temporary store with the array, so the per-FOV
temporary directory only needs to hold the Ultrack database again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(deps): pin iohub to main now that #486 merged

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(project)!: biahub project CLI; segment and track no longer project

Add `biahub project`, which turns a 3D plate into a 2D (Z=1) plate with one
projection per output channel: max, mean, sum over a Z range or a window
around a plane, or one plane (`slice`). The plane may be fixed or the in-focus
plane found with the registration engine's focus criterion (FocusSettings
subclass: centre crop, NA, wavelength), once per position (median over T) or
per timepoint. Ranges are half-open like concatenate's Z_slice and relative
to `center` when one is set, so the A549 60-plane window is
`center: focus, range: [-20, 40]`. Unlisted input channels are dropped with a
warning. The focus plane of every timepoint is recorded in the output
position's `biahub-project` metadata.

Segment and track stop selecting and projecting Z-planes. Segment loses
`z_slice_2D`: a model without do_3D segments in 2D and needs a Z=1 input.
Track loses `z_slicing` and the focus window; `output_mode` is checked
against the input Z. Both raise, pointing at `biahub project`, when a 3D
input meets a 2D request. Old keys are rejected with a migration message.

Nextflow: a project step between assemble and track (`5-project`,
`6-track` for A549); track reads the projected plate when project runs.
The A549 project config reproduces the previous tracking input (mean over
the 60-plane focus window on Phase3D), measured on the 800x800 centre crop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(nextflow): A549 project.yml projects every assembled channel; skill checks it

The A549 projection config lists every channel of a typical assembled plate
under the pre-rename names project sees inside the pipeline: the in-focus
plane of BF and Phase3D, the full-stack max of the raw fluorescence channels,
and the mean of the virtual-stain predictions over the 60-plane focus window.
The focus is found on Phase3D per timepoint, as biahub#400 asks for inference.
The brightfield key is `BF`, matching reconstruct.yml; the fluorescence names
and an older `BF - Oblique` are per-dataset edits.

The reconstruct-with-nextflow check script also validates project.yml both
ways (a listed channel the plate lacks is a MISS, an assembled channel the
config omits is NOT PROJECTED) and track.yml against the projected plate. The
skill carries unprojected channels into the pre-run plan for the user to
decide, runs the rename on the projected plate too, and suggests the
`BF - Oblique` edit in both reconstruct.yml and project.yml when the raw
store names it so.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(project): float32 plate for any max/mean/sum channel; drop float64 accumulation

NumPy already accumulates safely (float64 for integer input, pairwise float32
for float32 input), so mean and sum no longer force a float64 temporary. The
output plate is float32 whenever a channel uses max, mean or sum -- float32 in
and out is the common case -- and keeps the input dtype only when every
channel is a slice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(project): focus record keys are z and z_per_t

The focus record under the output position's biahub-project metadata is
focus: {channel, per, z, z_per_t}; the focus_ prefix repeated the block name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(project)!: z selection is start/stop or center with below/above

`range` meant absolute planes without a center and offsets with one. The two
forms now have their own keys: `start`/`stop` are absolute, half-open like
concatenate's time_indices; `below`/`above` count planes around `center`,
selecting [center - below, center + above), so the A549 window is
`center: focus, below: 20, above: 40`. The validator refuses mixing them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(nextflow): cleanup never targets the projected plate

The projected store is a deliverable like the assembled plate: cleanup
removes only its slurm_output/ and .iohub-progress/, as the wiring already
did; say so in the comment.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(project): chunks_czyx and shards_ratio; A549 writes 16-timepoint shards

ProjectionSettings takes the output chunk shape and shards ratio with
concatenate's names and semantics (shard = chunks * ratio, C not sharded) and
passes them to create_empty_plate; None keeps iohub's defaults.

The A549 config, following the Dynamic Cell Atlas array standard where a 2D
float32 plate can: one-frame 256 x 256 chunks, the grid the assembled and
tracking stores share, and shards of 16 timepoints by the full frame (the
standard's T-shard floor; ~150 MB each, 30 files per position instead of 400).
iohub writes one shard's timepoints per unit, so resume is 16 frames coarse.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* perf(project): find the focus of the timepoints in a process pool

The focus pre-pass ran the timepoints one after another while the job's
other workers idled; with per-timepoint focus on a long timelapse it was the
serial part of the step. The timepoints are independent, so with more than
one worker they now run in a spawned ProcessPoolExecutor sized by the job's
num_workers (the count the projection itself uses), each worker reading its
own volume. One worker keeps the in-process loop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(nextflow): projected plate lives beside the assembled one as <ds>_proj.zarr

The projected plate is the 2D companion of the assembled plate, not a step
of its own: the pipeline now writes it to <N>-assemble/<DATASET>_proj.zarr,
so project has no step directory and track stays 5-track. Cleanup already
covers the shared directory's slurm_output/ and .iohub-progress/; the
slurm_output README notes that a directory can hold two steps' stores. The
skill docs, run template and config comments follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Preserve focus metadata

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Validate existing output layout before reuse

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Include channel order in settings fingerprint

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* fix(project): size RAM by the channels read

iohub reads one timepoint's (C_read, Z, Y, X) block per worker, so the
request must grow with the listed channels: the fixed ram_multiplier of 4
gave 4 GB/CPU whether a config read 2 channels or 6, and the A549 config
(6 channels, 3.8 GiB per read, ~5-6 GiB peak) would have hit the cgroup.
ram_multiplier is now len(read_indices) + 2 (the read, the window copy and
the shard's output buffer): 6 GB/CPU for A549, unchanged for 2 channels.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(project): focus record carries the focus settings

Follows c8f8e78, which dumps the focus settings (minus axes) into the
biahub-project focus record next to z and z_per_t.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(project): more RAM headroom per worker, calibrated on an A549 position

One position of 2026_08_11 SEC61B v8 (6 channels, 86 x 1664 x 1193 float32,
5 shard units busy) peaked at 22 GB max RSS / 28 GB over the process tree,
~5-6 GiB per worker against the 6 GB/CPU requested. +4 volumes requests
7 GB/CPU there, a margin for a timelapse that keeps all 8 workers busy. The
time request (70 min for a 3.6 min run) is left as is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): ultrack pin follows royerlab/ultrack#290's pyupgrade fix

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(track)!: cellpose is the default segmentation method

TrackingSettings.segmentation_method defaulted to foreground_contour, the
legacy path no production config has used since April; a config that
omitted the key fell back to it silently. The default is now cellpose, and
cellpose without a cellpose_config is refused at load time instead of
falling back. The unit tests, which need no weights or GPU, name
foreground_contour explicitly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(pyramid): pyramids on images and label images; mantis-v2 adds them last (#359)

`biahub pyramid` is rewritten to the processing-command template and
computes pyramids on each position's image and on every label image under
labels/, in place: --levels (default 3, the DCA v0.2 target), --method for
the images and --label-method (max/stride/mode, default max) for the labels,
so no level holds a label id level 0 does not. Spatial axes of size 1 are
not downsampled, so the projected Z=1 plate gets factor 2 in Y and X only
and keeps its Z scale. --cluster/--monitor/--init replace --local; --init
refuses a plate carrying a pyramid of another depth and prints RESOURCES.
PyramidSettings validates the options and is the biahub-pyramid provenance
record on each position.

mantis-v2 gains --pyramid_levels (default 3, 1 = off) and a pyramid module
run once per plate, assembled and projected, as the LAST writer of each:
after tracking and after QC's compute, since all three rewrite the
position's zarr.json. Level 0 is never touched.

iohub is pinned to Biohub/iohub#488 (PositionLabel.compute_pyramid via
a shared MultiscaleNode; mode downsampler fix).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(nextflow): QC is the last step; pyramids run after tracking and ahead of it

QC's compute is gated on each plate's pyramid (or on tracking / the step that
created the plate when pyramids are off) rather than running concurrently
with projection and tracking, so imaging-qc's position-attr stamps never race
another writer of the same zarr.json and the report describes the finished
plates.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows the #488 review fixes (label-safe defaults, pyramid_dims)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows the second #488 review round

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows the third #488 review round

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(segment,track)!: segmentation moves out of track into the plate

The pipeline becomes assemble -> project -> segment -> track: `biahub segment`
writes one OME-NGFF label image per model into the plate it reads
(`labels/<model>`, no -o; -o keeps the separate label plate) through iohub's
`process_single_position(output_label=...)` (Biohub/iohub#489), so a
preempted position resumes. `biahub track` takes that label image as its
input (`labels: <model>`, `labels_sigma`) and writes the tracked labels beside
it as `labels/<model>_tracked`, with tracks.geff and the CSV. The cellpose
code, `cellpose_config` and `target_channel` leave track (both keys are
rejected with a pointer to the segment config); `foreground_contour` stays as
the legacy path and keeps `target_channel` and `input_images`. Blank frames
listed in the CSV are filled with the nearest frame's labels, as the images
were before. Track no longer needs a GPU: the `cpu` partition (a new
`cpu_reserved` Nextflow label; ultrack cannot resume).

mantis-v2.nf gains --segment_config between project and track; segment's init
creates the label images so track's init can check that its input exists, and
qc_project.yaml counts `labels/nuclei_tracked`. The A549 segment.yml takes the
cellpose values tracking was validated with (cpsam_v2, diameter 80, min_size
2500). Label chunking helpers move to biahub.utils.ngff.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(segment,track): scaffold labels with iohub's create_empty_labels; track on preempted

The label images are created by iohub's new `create_empty_labels`
(Biohub/iohub#489), which takes each label's geometry from its position
and records the provenance on the label group, so biahub's wrapper around
create_empty_plate goes; `label_layout` keeps only the DCA chunk sizing.

Track goes back to the `preempted` partition (the Nextflow `cpu` label):
ultrack cannot resume, but with segmentation out of track a position tracks
in a minute or two, so a preemption costs that much and the retry covers it.
The `cpu_reserved` label is dropped again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin moves to the #461 branch (pyramid levels keep level-0 shard bytes)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(settings)!: shards replaces shards_ratio in the concatenate and project configs

`shards` takes iohub's forms (Biohub/iohub#461): a size target such as
"5GB", "XY"/"XYZ", or an explicit TCZYX shape in array elements, so a config
no longer carries chunk arithmetic that must be redone per output shape. A
config that still says shards_ratio is refused with a message naming the new
key. Concatenate reads the shard's T extent back from the plate its init
created to size its workers. Track scaffolds its label images with
create_empty_plate(label_shards=...), an explicit (16, 1, Y, X) shape capped
at 2048 px, the same geometry as before.

Templates: a549 and zebrafish concatenate `shards: 5GB` (A549 unchanged at 5
timepoints by the full volume; dynatrack goes from 10 to 9 timepoints, under
the DCA 5 GB SHOULD); a549 project `shards: 200MB` (21 frames by the full
frame instead of 16).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(track): the tracked label image takes the source label's dtype as well

label_layout(source_label=...) copies dtype, chunks and shards; track writes
its ids in that dtype and refuses, rather than wraps, ids it cannot hold.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(segment,track)!: segment writes only into the plate; track's output_label

`biahub segment` loses `-o` and the separate label plate: its labels are the
plate's label images, one per model, and segmentation strategies accumulate
there as differently named models. A name is only rewritten by the model that
made it: `--init` refuses a label image whose recorded `biahub-segment`
settings (or shape) differ from the current model's, pointing at a new model
name. `output_ome_zarr_version` is rejected, since the labels take the
plate's version.

`biahub track` gains an optional `output_label`, another name for the tracked
label image, for several tracking runs of one segmentation side by side; then
the GEFF and CSV are `tracks_<output_label>.geff` / `tracks_<output_label>_<fov>.csv`,
while the default keeps `tracks.geff` / `tracks_<fov>.csv`, the names the
downstream readers expect.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(configs): concatenate chunks 1 x 8 x 256 x 256; iohub pin keeps level-0 chunks down the pyramid

The templates' float32 chunks go from 4 MiB to 2 MiB (Z 16 -> 8); with
shards: 5GB the A549 plate's shards become 6 timepoints by the full volume
(4.5 GiB). iohub dev/pyramid-shards now keeps the level-0 chunk shape at
every pyramid level, clamped to the level, instead of halving it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(track)!: the tracked label image is labels/tracks_<labels>

The default name of the tracked label image gains the `tracks_` prefix instead
of a `_tracked` suffix, so `labels/tracks_nuclei` sits beside `tracks.geff`
and `tracks_<fov>.csv`, and an explicit `output_label` names all three the
same way (`labels/<output_label>`, `tracks_<output_label>.geff`,
`tracks_<output_label>_<fov>.csv`). The A549 QC config counts
`tracks_nuclei`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(pyramid): --resume recomputes only the shards an interrupted attempt did not finish

Each shard of each pyramid level is a resumable unit in iohub
(czbiohub-sf/iohub feat/pyramid-resume, on #461): its files are cleared
before the write and a marker beside the plate records it once synced. The
token is the PyramidSettings fingerprint, so other levels or methods
recompute. The Nextflow module passes --resume unconditionally, like every
resumable step's. The iohub pin moves once that branch is on GitHub.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(pyramid): size the task from the level-0 shard, measured on an A549 position

iohub writes a pyramid level one target shard at a time and reads the source
region feeding it, so RAM follows the shard, not the volume: one assembled
A549 position (4.1 GiB level-0 shards, levels sized to the same bytes) peaked
at 44 GB and took 41.5 min against the previous 16 GB / 30 min request. The
estimate now asks 12x the shard bytes (52 GB there, the 16 GB floor on the
projected plate) and 0.15 min per (t, c) volume (70 min).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin moves to dev/pyramid-shards with the resume branch merged

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): name iohub#491 in the pin comment

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(pyramid): double a retried task's memory after an OOM kill

run_pyramid requested the init estimate verbatim on every attempt, so an
estimate that was too low failed all five attempts of every position with the
same 137 and aborted the run. The v9 debug run of 2026_08_11_A549_SEC61B_DENV
asked 16 GB 25 times and was OOM-killed 25 times. Use retry_memory like the QC
and virtual-stain processes: the next attempt doubles after a 137 and keeps
the same request after a preemption.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(project): double a retried task's memory after an OOM kill

run_project asked for its init estimate verbatim on every retry, so a low
estimate failed all attempts with the same 137. Use retry_memory like the other
SLURM steps (fix/nextflow-retry-memory-cap carries the 244 GB cap).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(pyramid): request 14x the shard and 0.3 min per volume, from v9's 54 positions

The v9 debug run of 2026_08_11_A549_SEC61B_DENV measured every position of
the assembled plate (level-0 shards 4.51 GiB): 46.5 GB peak RSS each (10.3x
the shard) and up to 67.3 min, the slowest on the 120 GB cpu-c nodes. The old
12x / 0.15 asked 56 GB / 70 min, 4% above the slowest position. 14x the shard
and 0.3 min per volume ask 64 GB / 130 min there; the projected plate keeps
the 16 GB floor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(project): max keeps the input dtype; only mean/sum make the plate float32

max picks an existing voxel, so promoting a max-only plate to float32
doubled its storage for nothing and would truncate uint32 labels above
2^24. A store has one dtype, so a plate with any mean/sum channel is
still float32 throughout and its max/slice channels are widened on write.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): pin iohub to main at the #491 merge (80b20a6)

One pin for the whole chain: main now carries #489 (output_label,
create_empty_labels), #488, #461 and #491, so the 16cdc4f branch pin and
the dev/pyramid-shards combination branch retire.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(segment,track)!: label images are (T, 1, Z, Y, X)

A label image now has the image's axes with one channel rather than TZYX:
the DCA array standard wants every array 5D, and OME-NGFF's own layout note
reads a label's dimensions against the image's. iohub creates and writes
them that way (czbiohub-sf/iohub, label channel axis); here the DCA label
chunks become (16, 1, 1, 128, 128), segment's shape check and track's label
reads and writes index the channel axis, and the tests and skill notes
follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): pin iohub to Biohub/iohub#496 (labels are (T, 1, Z, Y, X))

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows Biohub/iohub#496's review fixes (9f21979)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): iohub pin follows Biohub/iohub#496 (eee7567)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): pin iohub to main at the #496 merge (0cb3e35)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* refactor(labels): size label shards with iohub's 200 MB target

`label_layout` passes iohub `label_shards="200MB"` (Biohub/iohub#461:
the largest chunk-aligned shard under the target, filled X, Y, Z then T, C
never grown) instead of a computed `label_shards_ratio`, which iohub has
deprecated. On the A549 projected plate the result is the same geometry as
before, one 16-timepoint chunk by the full frame; #430's segment labels use
the same constant, so the two branches agree.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* build(deps): the iohub override follows the pin (0cb3e35)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* style: ruff format

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant