Skip to content

perf: read OME-Zarr planes through one map_blocks layer (0.5 s -> 55 ms per slider step) - #45

Merged
hongquanli merged 2 commits into
mainfrom
perf/zarr-direct-plane-reads
Nov 8, 2026
Merged

hongquanli merged 2 commits into
mainfrom
perf/zarr-direct-plane-reads

Conversation

@hongquanli

Copy link
Copy Markdown
Collaborator

Summary

Moving a slider on a large OME-Zarr plate took 0.3–0.65 s per step. Almost none of that was reading pixels: dask was re-planning a huge task graph for every plane.

The loader built the 6-D array by opening each field as its own dask array (da.from_array over its tensorstore array) and then da.stack/moveaxis (plate, per-FOV) or da.concatenate (6D regions). On a real Squid plate (1772 fields × 28 z, 2084 × 2084 uint16) the result has 3547 graph layers and 49 616 chunks. For each plane ndv requests, dask orders and optimises that graph (dask.order, optimize, legacy-graph conversion): 0.5–1 s, against ~20–30 ms for the tensorstore read itself.

This builds the (time, fov, z, channel, y, x) array with a single map_blocks over a dummy array instead, the same pattern the single-TIFF loader already uses. Each block maps its fov index to (image, local field) and reads its (y, x) plane straight from that image's tensorstore array.

Unchanged

  • Every field is still opened at load, so a broken field still fails the load with the same cannot open … error.
  • Axis validation still goes through _arrange_image_axes (run on a zero-memory stand-in with the array's shape), so the error messages that name the array are the same. Axes are still arranged by name, including x stored before y.
  • 6D: same region-shape check and message. Plate/per-FOV: a field whose shape differs from the first now raises OmeZarrError naming that field; before, da.stack raised a generic ValueError.
  • _TensorStoreArrayWrapper is removed; nothing else used it.

Numbers

Real plate above, Windows, PyQt5, ndv 0.4.1 (main), real window. A "step" is one slider move until the new plane is on screen.

before after
slider step (fov / z) 650 / 300 ms 57 / 57 ms
frames drawn during a 2 s drag (fov, a move every 16 ms) 4 / 20 61 / 62
opening the plate 5.2 s 3.9 s

With #44 (ndv 0.5) plus its two follow-up commits, the same plate steps in ~33 ms.

Tests

tests/test_zarr_plane_reads.py. Each field and position gets its own pixel values, because the existing fixtures write the same values into every field and would not catch a wrong-field read.

  • every plane of a 2-well / 5-field plate matches its own field;
  • 6D: two regions concatenate with the right (region, local field) for each fov;
  • x stored before y is transposed;
  • the graph has the same number of layers for 1 and 12 fields (fails on main: it grows with the field count).

Full suite: everything passes except the two tests that already fail on Windows on main, because they expect / in paths (test_existing_field_without_metadata_is_broken, test_broken_fov_fails_the_load). black --check . is clean.

🤖 Generated with Claude Code

Loading a plate stacked one dask array per field (da.from_array over each
field's tensorstore array, then da.stack/moveaxis or da.concatenate). On a
real Squid plate (1772 fields x 28 z, 2084 x 2084 uint16) that graph has
3547 layers and 49616 chunks, and dask spends 0.5-1 s ordering and
optimising it for every plane ndv requests, against ~20-30 ms for the
tensorstore read itself.

Build the (time, fov, z, channel, y, x) array with a single map_blocks over
a dummy array instead, as the single-TIFF loader already does: each block
maps its fov to (image, local field) and reads its (y, x) plane straight
from that image's tensorstore array. Axis validation, the shape checks and
their error messages are kept; axes are still arranged by name, including
x stored before y.

On that plate: slider step 300-650 ms -> ~55 ms, opening 5.2 s -> 3.9 s.

Tests: every plane comes from its own field (plate, 6D regions, x-before-y
storage), and the graph size does not grow with the number of fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the dask-version layer count

Review of #45 at c0fb5e3, all three reproduced against main (76585da):

1. Mixed field dtypes: the plane array took the first image's dtype, while
   da.stack / da.concatenate promoted over every field. A uint8 first field
   beside a uint16 one gave a uint8 array whose blocks still arrived as
   uint16 (60000 read as 96 by anything that honours the declared dtype).
   _zarr_common_dtype() is np.result_type over every image's dtype, used by
   the plate/per-FOV and the 6D loaders, and load_block casts each plane to
   it.

2. Images missing y or x: the array shape came from _arrange_axes on a
   stand-in (singleton for the missing axis, as before), but load_block
   indexed dim_names for "x" and "y" unconditionally, so a (t, c, z, y) image
   loaded and then raised "'x' is not in list" on the first read; on main it
   read an (8, 1) plane. load_block now arranges the plane's spatial axes
   with the same _arrange_axes call, which also replaces the x-before-y
   transpose.

3. test_graph_size_does_not_grow_with_fields asserted <= 2 layers; dask
   2024.9 builds three (block-info, zeros_like, load_block) for any field
   count and the test failed there although the graph is constant. It now
   asserts the layer count is the same for 1 and 12 fields and that the
   partition count scales with them.

tests: mixed-dtype plate and 6D regions (array dtype uint16, every block
uint16 with its own value), t c z y and t c z x images (sizes, plane shape,
pixels), fixture write_image_group(dtype=) passthrough. Full suite 294
passed under PyQt5 and PyQt6 (ndv 0.4, dask 2024.9).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@hongquanli
hongquanli merged commit 0734d8c into main Nov 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant