Repository navigation
perf: read OME-Zarr planes through one map_blocks layer (0.5 s -> 55 ms per slider step) - #45
Merged
Merged
Conversation
Loading a plate stacked one dask array per field (da.from_array over each field's tensorstore array, then da.stack/moveaxis or da.concatenate). On a real Squid plate (1772 fields x 28 z, 2084 x 2084 uint16) that graph has 3547 layers and 49616 chunks, and dask spends 0.5-1 s ordering and optimising it for every plane ndv requests, against ~20-30 ms for the tensorstore read itself. Build the (time, fov, z, channel, y, x) array with a single map_blocks over a dummy array instead, as the single-TIFF loader already does: each block maps its fov to (image, local field) and reads its (y, x) plane straight from that image's tensorstore array. Axis validation, the shape checks and their error messages are kept; axes are still arranged by name, including x stored before y. On that plate: slider step 300-650 ms -> ~55 ms, opening 5.2 s -> 3.9 s. Tests: every plane comes from its own field (plate, 6D regions, x-before-y storage), and the graph size does not grow with the number of fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the dask-version layer count Review of #45 at c0fb5e3, all three reproduced against main (76585da): 1. Mixed field dtypes: the plane array took the first image's dtype, while da.stack / da.concatenate promoted over every field. A uint8 first field beside a uint16 one gave a uint8 array whose blocks still arrived as uint16 (60000 read as 96 by anything that honours the declared dtype). _zarr_common_dtype() is np.result_type over every image's dtype, used by the plate/per-FOV and the 6D loaders, and load_block casts each plane to it. 2. Images missing y or x: the array shape came from _arrange_axes on a stand-in (singleton for the missing axis, as before), but load_block indexed dim_names for "x" and "y" unconditionally, so a (t, c, z, y) image loaded and then raised "'x' is not in list" on the first read; on main it read an (8, 1) plane. load_block now arranges the plane's spatial axes with the same _arrange_axes call, which also replaces the x-before-y transpose. 3. test_graph_size_does_not_grow_with_fields asserted <= 2 layers; dask 2024.9 builds three (block-info, zeros_like, load_block) for any field count and the test failed there although the graph is constant. It now asserts the layer count is the same for 1 and 12 fields and that the partition count scales with them. tests: mixed-dtype plate and 6D regions (array dtype uint16, every block uint16 with its own value), t c z y and t c z x images (sizes, plane shape, pixels), fixture write_image_group(dtype=) passthrough. Full suite 294 passed under PyQt5 and PyQt6 (ndv 0.4, dask 2024.9). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moving a slider on a large OME-Zarr plate took 0.3–0.65 s per step. Almost none of that was reading pixels: dask was re-planning a huge task graph for every plane.
The loader built the 6-D array by opening each field as its own dask array (
da.from_arrayover its tensorstore array) and thenda.stack/moveaxis(plate, per-FOV) orda.concatenate(6D regions). On a real Squid plate (1772 fields × 28 z, 2084 × 2084 uint16) the result has 3547 graph layers and 49 616 chunks. For each plane ndv requests, dask orders and optimises that graph (dask.order,optimize, legacy-graph conversion): 0.5–1 s, against ~20–30 ms for the tensorstore read itself.This builds the
(time, fov, z, channel, y, x)array with a singlemap_blocksover a dummy array instead, the same pattern the single-TIFF loader already uses. Each block maps its fov index to(image, local field)and reads its(y, x)plane straight from that image's tensorstore array.Unchanged
cannot open …error._arrange_image_axes(run on a zero-memory stand-in with the array's shape), so the error messages that name the array are the same. Axes are still arranged by name, including x stored before y.OmeZarrErrornaming that field; before,da.stackraised a generic ValueError._TensorStoreArrayWrapperis removed; nothing else used it.Numbers
Real plate above, Windows, PyQt5, ndv 0.4.1 (main), real window. A "step" is one slider move until the new plane is on screen.
With #44 (ndv 0.5) plus its two follow-up commits, the same plate steps in ~33 ms.
Tests
tests/test_zarr_plane_reads.py. Each field and position gets its own pixel values, because the existing fixtures write the same values into every field and would not catch a wrong-field read.Full suite: everything passes except the two tests that already fail on Windows on main, because they expect
/in paths (test_existing_field_without_metadata_is_broken,test_broken_fov_fails_the_load).black --check .is clean.🤖 Generated with Claude Code