Repository navigation
perf(j2k): stream large decodes and bounded rows through pool bands - #316
Merged
Merged
Conversation
U8RowSink packs three 8-bit component rows with one indexed loop over RGB triples (round and clamp to 0..=255, as sample_as_u8 does for 8 bits) instead of a per-sample call that branches on the bit depth. Multi-threaded region decodes run this pack serially after the IDWT. Measured together with the byte-run reads of stuffed packet-header bits, on an 18-core ARM64 machine, 5 interleaved rounds, median of per-round ratios, on regions of the 18000 x 12000 PLT test image (where packet parsing is small): top-left -11%, middle -8.5% and bottom-right -9% on 18 threads, middle -6.6% on 6 and -1.3% on one.
…l bands A parallel region decode on the row-streamed IDWT decoded one component's bands on the pool while that component's IDWT ran, then the next component's, and wrote component planes. At reduced resolutions each sub-band holds few bands, so one component's bands left most of the pool idle, and classic reduced-resolution regions decoded more slowly at 18 threads than before the band pool. Now a classic tile whose rows stream to a row sink runs the bands of all its components on one executor, and the IDWT advances the components together, a row of each at a time, handing the rows to the sink as the serial bands of `banded_tier1::decode_tile_rows` do. The region needs no component planes. - The components' rings and bands are planned together, and workers take the band the IDWT reads first across all components. - `streams_rows` accepts tiles that decode their bands on the pool, and the line buffer charged for lockstep rows covers them. - A tile the route declines decodes its bands serially; one stranded before any row streams does too. A failure reruns whole-window Tier-1 serially for the error, as before. - HTJ2K tiles keep component planes: their bands decode fast enough that the planes' inverse transform on the pool costs less than transforming rows on the calling thread. Measured with region decodes to RGB8 (5 interleaved rounds, median of per-round ratios; output byte-identical at 1, 6 and 18 threads) on regions of an 18000 x 12000 RGB image: classic with PLT markers, classic without, and HTJ2K. Against the parent: - Classic whole image at 1/16 scale: -29% at 6 threads, -53% at 18; at 1/8 scale: -9% and -47%. Against the commit before the band pool, the 18-thread rows are 20% and 30% faster. - Classic PLT 2048 x 2048 at 1/2 scale: -11% at 6 threads, -22% at 18. - Classic PLT 1024 x 1024: -3% to -5% at 6 threads, 0% to -5% at 18; without PLT -2%. - HTJ2K rows: -1.2% to +0.1%. - Peak heap: -29% to -59% on the classic rows at 6 and 18 threads, which hold no component planes; HTJ2K within +0.2%.
…ernel The row-streamed IDWT interleaved each row's low- and high-pass samples into a row and lifted it with the scalar filter. It now runs the banded IDWT's SIMD horizontal synthesis on the stored band rows directly. The kernel applies the same operations in the same order, so the output is bit-identical. - A row whose band rows lie wholly inside their stored windows reads them in place; a band row wholly outside reads zeros. Rows that straddle a window edge, rows narrower than the SIMD kernel's minimum, and levels one row tall keep the scalar path. - The line rings carry one row of horizontal scratch for the widest level, charged with them. - `synthesize_row` takes the level's row geometry instead of the whole level, so the streamed rows can call it. A test compares one- and two-level streamed rows with the whole-window synthesis, bit for bit, for both wavelets, odd and even origins, and windows clipped on every side. Measured with region decodes to RGB8 (5 interleaved rounds, median of per-round ratios; output byte-identical at 1, 6 and 18 threads) on regions of an 18000 x 12000 RGB image: HTJ2K, classic with PLT markers, and classic without. Against the parent / against f33e40c: - HTJ2K whole image at 1/4 scale: -52% / -53% at 6 and 18 threads; at 1/16 scale: -36% / -36% at 6 threads, -34% / -35% at 18. - HTJ2K 1024 x 1024: -29% / -30% at 6 threads, -27% / -28% at 18. - Classic PLT 1024 x 1024: -16% to -28% / -20% to -30% at 6 threads, -32% to -36% / -34% to -37% at 18. - Classic PLT 2048 x 2048 at 1/2 scale: 0% / -11% at 6 threads, -26% / -42% at 18. - Classic whole image at 1/8 and 1/16 scale: -1% to -2% / -10% to -30% at 6 threads, -6% to -15% / -54% to -56% at 18. - One thread, the serial row path the parent leaves alone, against f33e40c: classic 1024 x 1024 -5% to -9%, 2048 x 2048 at 1/2 scale -4%, whole image at 1/16 scale -2%; HTJ2K 1024 x 1024 -27%, whole image at 1/8 scale -35%.
…8 in place A tile streaming rows from pool bands reconstructed every component on the calling thread, a row at a time, transformed each row and handed it to the sink, which packed it. The calling thread's IDWT and pack were the decode's critical path once Tier-1 ran on the pool. - Rows reconstruct in batches of up to 256 rows, 256 Ki samples per component (`streamed_batch_rows`), charged with the line buffer. - Each component's reconstruction advances through its own view of the bands, on the pool, when the tile is HTJ2K, or classic with 12 pool threads or more; with fewer, the classic readers would hold threads the bands need, and one view on the calling thread reads every component. Each view takes a workspace from the bank, so the route sends that many fewer workers. - A single classic view that waits for a band a worker is decoding first decodes the next band nobody has started, counted as an active worker while it does. - Three 8-bit components pack straight into the sink's interleaved RGB8 rows when the new rustdoc-hidden `ComponentRowSink::rgb8_region` offers them, which the facade's 8-bit sink does for tight RGB8 output without alpha. Other sinks take one `write_rows` call per batch. - HTJ2K tiles now stream rows from pool bands too. - A view that waits on a band whose worker failed returns the shared error and leaves it for the other views. Tests compare pool rows with component planes at 2, 4 and 12 threads, corrupt code-blocks through several views, and the RGB8 destination with the sink's own pack, byte for byte. Measured with region decodes to RGB8 (5 interleaved rounds, median of per-round ratios; output byte-identical at 1, 6 and 18 threads) on regions of an 18000 x 12000 RGB image: HTJ2K, classic with PLT markers, and classic without. Against the parent / against f33e40c: - HTJ2K whole image at 1/4 scale: -35% / -78% at 6 threads, -32% / -78% at 18; at 1/16 scale: -26% / -63% at 6, -30% / -62% at 18. - HTJ2K 1024 x 1024: -16% / -47% at 6 threads, -20% / -48% at 18. - Classic at 18 threads (a reader per component): PLT 1024 x 1024 -9% to -18% / -46% to -54%; whole image at 1/16 scale -12% / -63%; PLT 2048 x 2048 at 1/2 scale +1% / -43%; whole image at 1/8 scale -1% / -56%. - Classic at 6 threads (one reader): PLT 1024 x 1024 -5% to +1% / -23% to -33%; whole image at 1/8 scale -10% / -19%; at 1/16 scale +5% / -27%. - One thread decodes serially, unchanged. - Peak heap of the parallel rows: -14% to -66% against f33e40c (no component planes); at one thread unchanged.
With a reader per component, the calling thread transformed and packed each batch alone while the other readers' threads waited for the next batch. The batch's rows now split into one range per reader, the first finished here and the rest spawned on the pool, each transforming its rows one at a time as before and packing them into its own output rows. One reader keeps a single range. Measured with region decodes to RGB8 (5 interleaved rounds, median of per-round ratios; output byte-identical at 1, 6 and 18 threads) on regions of an 18000 x 12000 RGB image: HTJ2K, classic with PLT markers, and classic without. Against the parent / against f33e40c: - HTJ2K whole image at 1/4 scale: -10% / -81% at 6 threads, -15% / -81% at 18; at 1/16 scale: -4% to -5% / -65% at 6 and 18. - HTJ2K 1024 x 1024: -2% / -48% at 6 threads, -4% / -49% at 18. - Classic at 18 threads: PLT 1024 x 1024 -3% to +2% / -47% to -57% (a 7-round recheck of the 1024 x 1024 rows: -1% to -2%); whole image at 1/8 and 1/16 scale -1% / -57% and -63%. - Classic at 6 threads, one reader: within +-1%.
Each batch of streamed rows ended in a join: the readers reconstructed it, then it was transformed and packed while every reader waited. With a reader per component, the tile now holds two batches: one is transformed and packed on the pool while the readers reconstruct the next into the other. Batches hold up to 2 Mi samples of each component and 512 rows, and at most a quarter of the tile's rows, so the two batches stay half of the rows' samples or less; fewer, larger batches spend less time in joins. One reader reconstructs a row of each component at a time within a batch, as the row-at-a-time path does, instead of a batch of one component after another, which left the other components' band rings full and the workers idle. Measured with region decodes to RGB8 (5 interleaved rounds, median of per-round ratios; output byte-identical at 1, 6 and 18 threads) on regions of an 18000 x 12000 RGB image: HTJ2K, classic with PLT markers, and classic without. Against the parent / against f33e40c: - HTJ2K whole image at 1/4 scale: -11% / -83% at 6 threads, -29% / -87% at 18; at 1/8 scale: -6% / -79% at 6, -18% / -81% at 18. - HTJ2K 1024 x 1024, 4096 x 4096 at 1/4 scale and whole image at 1/16 scale: -4% to +0.1% / -48% to -69%. - Classic PLT whole image at 1/16 scale, one reader at 6 threads: -12% / -36%. - Other classic rows at 6 and 18 threads: -2% to +2% / -13% to -64%. - Peak heap against f33e40c: -48% to +0.2% at 6 and 18 threads (the +0.2% on 4096 x 4096 at 1/4 scale), at most +0.1% at one thread.
A classic tile whose components reconstruct on their own readers waits on classic Tier-1: at 18 threads its pool runs out of bands to start while the rings are full. Such a tile now holds one band more in every ring, and one batch of reconstructed rows instead of two, which more than pays for the bands. HTJ2K tiles keep the shallower rings and the second batch, which their readers use. Measured with region decodes to RGB8 (5 interleaved rounds, median of per-round ratios; output byte-identical at 1, 6 and 18 threads) on regions of an 18000 x 12000 RGB image: classic with PLT markers, classic without, and HTJ2K. Against the parent / against f33e40c, at 18 threads: - Classic PLT 1024 x 1024: -4% to -6% / -49% to -56%. - Classic whole image at 1/16 and 1/8 scale: -4% / -65% and -57%. - Classic 2048 x 2048 at 1/2 and 4096 x 4096 at 1/4 scale, whole image at 1/4 scale, and 1024 x 1024 without PLT: -1% to +0.1% / -25% to -56%. - At 6 threads and for HTJ2K: within +-1%. - Peak heap: -2% to -11% against the parent on the classic rows at 18 threads; at most 0.98x of f33e40c's on any row at 6 and 18 threads.
`write_rgb_component_rows_u8` rounded each sample with `f32::round`, which is a `roundf` call per sample on x86-64 and does not vectorize there. Rows whose three planes are 8-bit now go through `j2k_native::pack_rgb8` (exported `#[doc(hidden)]`), which packs at the CPU's SIMD level on x86-64; ARM64 keeps its auto-vectorized loop. Output is byte-identical. On x86-64 this commit was timed only together with the block-load commit and a region row-filter dispatch that was not kept, so its own share of the region and overview gains is not isolated.
…ing them A prepared item whose offset-plan execution exceeds the per-decode bound (the 5661 x 8287 RGB8 scan: 563 MB of planes; RGB from 4096 x 4096, gray from 6656 x 6656) failed with ImageTooLarge although the facade's full-output region route decodes it in bounded memory. Preparation now marks such Gray8/RGB8 items for streamed execution, a choice made from the plan before any decode work. They decode through the region rows route decode_region_into uses (whole output rectangle, at the requested scale) into a row sink that writes the batch's NCHW/NHWC samples with the same rounding as whole planes, and claim one worst-case native decode of admission while in flight (no planned figure for that route yet). Groups holding one never take the staged route. Other outputs keep the typed prepare rejection. Items that fit keep their plans, bytes and times.
A homogeneous group's dense output larger than the per-allocation cap (eight 5792^2 RGB8 images, 805 MB) failed the whole decode call with AllocationTooLarge. Such a group now decodes as several groups with the same info and consecutive images, each buffer within the cap. Groups that fit are unchanged; an image whose own output exceeds the cap still fails.
…n parallel Parallel referenced classic dispatch borrowed every payload as one input slice and fell back to serial code-block decode when a block's bytes were split over fragments. Multi-layer codestreams (one fragment per contributing layer) therefore decoded every block on one thread; only the wavelet steps ran in parallel. Row jobs now carry their payload's fragment ranges, and each row task joins fragmented payloads in its own buffer: the main task in the scratch's combine buffer, every extra task in a buffer kept beside its code-block workspace. Each task buffer is planned and reserved at the plan's largest fragmented payload (none when every payload is one input slice), so planning, admission and the retained-scratch accounting see it. Every block decodes exactly as on the serial path; samples are bit-identical.
…e planes Every Gray8/RGB8 image on the CPU batch decoder's offset-plan route held whole-image f32 planes and coefficients, about 14 times its 8-bit output, until its decode ended: eight 3584^2 RGB8 images in flight peaked at 4.3 GB for 294 MB of output, with one worker as with the pool (the staged route keeps every window image's scratch). Preparation now marks a Gray8/RGB8 image whose planned execution exceeds 4 MiB for the streamed region rows route that over-budget images already use, which holds a band of rows instead of planes; its output is bit-identical. Groups with such an image skip the staged route, so a batch's peak is the streamed working sets of the images in flight plus the outputs. Images below the threshold, and outputs without a streamed route (16-bit, RGBA), keep their parse-free plans. The one exception is a lone Gray8 image (the only image of its group, hence the whole in-flight set) whose plan fits the per-decode bound: it keeps decoding its plan, which spreads one component's code-block rows over every core; streamed, such images decoded 26-66% slower. RGB8 images stream faster than their plans decoded (whole image, reduced scales, regions, tiles). An explicit single worker over several 8-bit images loses the staged route's cross-image windows: about 30% slower there for a tenth of the memory.
A tile whose coarsest resolutions hold no samples while finer ones do (a one-sample tile row or column at an odd position, two or more levels) has no IDWT window for those levels. Both region IDWTs took any level without a window as nothing to reconstruct and gave the component up: the whole-window path cleared it (mid-gray planes for 16-bit and component-plane regions) and the streamed line plan returned no rows (IncompletePlaneStore or CodeBlockDecodeFailure for 8-bit regions). They now skip the leading levels whose decomposition rect is empty and start at the first level with a window, from an empty LL band, as the full-image IDWT does. Changes output where decodes were wrong: over 632 region cases of tiles with empty coarse levels (8- and 16-bit), the decoder gave 187 wrong-pixel results and 316 IncompletePlaneStore/CodeBlockDecodeFailure errors; now all decode and pass the region-vs-full-crop check, or return the Unsupported colour-space error they returned before. Full-image decodes are unchanged.
…ecode The tile and band routes copied the payload of every block whose segments span several packets into one arena per route before any block decoded, serially on the calling thread. A full-image streamed decode routes every block of the tile, and with several layers that is every payload, so the arenas held the tile's coded bytes for the whole decode: 17.4 MiB on a 5661x8287 RGB image with 8 layers, 57.6 MiB on an 18000x12000 one. Each block now gathers its segments from the tile's layer and segment records into its thread's scratch just before it decodes, through a borrowed `PayloadSource`; the scratch grows to the largest gathered payload and is charged per thread instead of the arena. Full-ROI streamed decode, 18 threads, heap above the output: 102.8 -> 85.6 MiB (5661x8287, 8 layers), 286.7 -> 229.2 MiB (18000x12000); at 1/2 scale 55.9 -> 44.4 MiB. Time unchanged within noise (110.6 -> 109.3 ms at 18 threads, 1365 -> 1364 ms at one thread). Output is identical.
…old too few bands A streamed decode's pool can only run as many band decodes as the finest level's rings hold bands ahead of the readers. With one component that is two bands in each of three rings, so six of eighteen threads decode Tier-1 at the finest level while the reader decodes the bands it waits for itself: a 5792x5792 gray image streamed in 158 ms against 63 ms for the component planes. When the finest rings hold fewer bands ahead than the executor has threads, each classic band now splits into runs of its blocks of about equal work (`split_runs`, as the tile route splits its strips): enough parts per finest band for two parts per thread (ceil(2 x threads / bands ahead), at most 8), so a thread that finishes a part finds another while a band's last parts finish, and parts of about the same work at the other levels. Threads decode a band's parts at once into its slot, each zeroing its own columns and copying its blocks in under the slot's lock (the tile route's `decode_part`); the last part to finish hands the band over. A ring's slots are cut into whole rows of its sub-band, at least as many as its longest band's, since a part zeroes its columns of the slot row by row. A reader waiting for a running band decodes its unstarted parts. RGB at 18 threads with today's rings has bands enough and plans no parts. Full-ROI streamed decode at the default pool (18 threads): 5792 x 5792 gray 158.0 -> 61.2 ms, 8192 x 8192 gray 322 -> 134 ms; on 6 threads 5792 x 5792 gray 163 -> 135 ms. RGB unchanged (5661 x 8287 112.8 -> 108.5 ms, 4096 x 4096 54.6 -> 52.8 ms, within noise). One part per thread instead of two left the 5792 x 5792 gray image at about 80 ms. Output is identical; new tests force splits of 2, 3, 5 and 7 parts against whole-window Tier-1 and the serial component planes, also for regions that clip the only code-block rows that code anything.
…he rows planned A tile keeps a layer record for every layer of every code-block it reads, 24 bytes each (`Option<Range<usize>>`): 6.5 MiB for a 5661x8287 RGB image with 8 layers, 25.6 MiB for an 18000x12000 one. A record is now a u32 start and length into the tile's segment records, with an unread layer marked by a length of u32::MAX; a range that does not fit is a packet parse error. The tile and band routes also reserved one strip per block and kept it; the strips now shrink to the code-block rows planned (1.5 and 5.3 MiB). Full-ROI streamed decode heap at the default pool: 5661x8287 85.6 -> 79.8 MiB, 18000x12000 229.2 -> 207.1 MiB; one thread 25.8 -> 21.4 MiB. Time unchanged. The layer-metadata amplification test now rewrites its header to 8192^2: at 8 bytes per record the 4096^2 header fits the decode budget and fails in packet parsing instead of being refused up front.
…d rings A classic tile whose components reconstruct on their own readers, at 12 pool threads or more, holds deeper band rings and a single batch of reconstructed rows, which it transforms before the next batch reconstructs. Nothing overlaps that batch, so a shorter one costs no speed: such a tile now reconstructs half as many rows per batch, as long as that leaves at least 128 rows. Tiles with two pipelined batches (HTJ2K, and classic tiles at fewer threads) keep their batches, as do rows wider than 8192 samples: halving an 18000-wide tile's 116-row batches cost 2 to 5%, and halving every tile's batch cost 5% on a 9000-wide HTJ2K overview. Measured with full-image region decodes to RGB8 (interleaved rounds, median of each run; output byte-identical) on an 18-core ARM64 machine, against the parent: - 5661 x 8287 RGB, one 9/7 tile: 109.6-112.5 ms before and after at 18 and default threads; peak heap 102.8 -> 90.8 MiB (-12.0 MiB, -11.7%). At 1/2 scale 55.9 -> 51.3 MiB at equal speed. At 6 and 1 threads the rings are not deep and nothing changes. - 4096-wide classic RGB mosaics (2bpp and lossless) and a 4096-wide classic RGB image at default threads: equal speed; peak heap 56.7 -> 44.7 MiB and 59.2 -> 47.2 MiB (-21%, -20%). - HTJ2K mosaics, a 9000-wide HTJ2K overview and an 18000 x 12000 classic image keep their batches. - One decode per fresh process against the code before the streamed full routes, at 1, 6, 18 and default threads: the 5661 x 8287 image at full, 1/2, 1/4 and 1/8 scale and the four mosaics run 0.98-1.04x (the noise of paths this leaves alone); the 5661 x 8287 image's footprint (max RSS less input and output) at 18 threads drops from 104.7 to 92.8 MiB.
…ed region A full-image decode keeps every component in an image-sized f32 plane, and the native decoder refuses one whose planes and largest tile's workspace pass its budget, so `decode_into` and `decode_scaled_into` of large images failed with a resource-limit error while a region decode of the same pixels streams its rows and fits. An 8-bit full-image decode now asks the native decoder, before decoding, what the planes route would charge, and runs as a region decode of the whole image when that passes the budget. The new hidden `Image::full_decode_planes_floor` plans the charge from the parsed tiles without allocating planes or coefficients: the tiles' structural bytes, every plane as `reset` sizes it, and the largest tile's records, coefficients and IDWT workspace as `build` plans them. Each term is at most what the planes decode charges, so an image the planes decode finishes is never rerouted. It returns None (keep the planes) where the whole-image region would not stream its rows, and for tiles whose coarse resolutions are empty while finer ones are not, which the region IDWTs do not reconstruct yet. A header-only estimate skips the tile parse for images far below the budget. `J2kDecoder` takes the budget from one helper (the native default today); a hidden setter lowers it for tests. Against the planes route's choice (the route disabled) over 26 test images of growing size at 1/1 to 1/8, 41 cases route and 63 keep the planes, none disagreeing, and the routed outputs equal the planes path's with its budget raised. Over 87 boundary-scan sizes no image that decodes today is rerouted; 11 within 1.5% of the budget still fail, as before (the plan leaves out Tier-1 workspaces and, with tiles, earlier tiles' retained scratch). The 5661 x 8287 RGB workload image decodes at full resolution with 102.8 MiB peak heap on the default pool (25.8 MiB on one thread) against 983.7 MiB (952.3) for the planes path with its budget raised, in 132 ms against 123 ms (1367 against 1345 ms on one thread). Untiled RGB images from 5792^2 to 16384^2 take 2-8% longer than that path with 6-17x less heap; 1024^2-tiled RGB 35-45% longer and grayscale 2.4-2.8x as long, the streamed route's own speed.
…ster A full-image 8-bit decode now runs on the streamed route not only past the decode budget but wherever streaming was measured at least as fast, chosen before decoding from the header and tiles (no rerun after a failure): - reduced-resolution decodes (decode_scaled_into) of untiled three-component classic images at any thread count: on the 5661 x 8287 workload at 1/2, 1/4, 1/8, 5-47% faster warm and 22-64% faster cold on 6 threads or more, equal on one, with 4-13x less heap; - full-resolution untiled ones of a megapixel or more on 2 to 11 pool threads (about 10% faster warm on 6 threads, 6-12% faster cold); from 12 threads on, where classic tiles reconstruct on pool readers, the planes stay 3-6% faster; - images with a busy tile for every worker, whose tiles now decode at once (new StreamedTiles route): the calling thread and one scoped thread per pool thread each region-decode one tile on their own thread into a tile buffer and copy it out. 4096 x 8192 RGB in 1024 x 1024 tiles: 99.6 vs 135.7 ms on the planes at 18 threads; over-budget 8192^2 RGB / 16384^2 gray in 1024^2 tiles: 242 -> 189 ms, 1350 -> 470 ms. Gray (single-component band lookahead), HTJ2K (5-50% slower streamed), small images and few-tile images keep the planes. The test hook set_full_decode_planes_budget now fixes the budget and drops the default preference, so tests can force either route. New doc-hidden native API: Image::output_tile_rects, cpu_decode_pool_threads.
…ges on the planes Refines the default full-image route from measurements on an 18-core ARM64 machine (interleaved, warm / cold medians): - reduced-resolution decodes of untiled three-component HTJ2K images now stream on two pool threads or more: a 4096 x 4096 HTJ2K RGB image at 1/2 and 1/4 takes 21% and 47% less time warm on 6 threads (cold 48% and 70% less), 26% and 53% less on the default pool; the 4096 x 2048 HTJ2K mosaic at 2 bpp at 1/2 and 1/4 9% and 35% less on 6 threads. One thread keeps the planes (1/2 is 2-9% faster there). - full-resolution untiled classic images stream on 2 to 11 pool threads only from 4 coded bits per pixel (coded bytes = the decoder's input): the 2.1 bits-per-pixel mosaic was 1-3% slower streamed on 6 threads (32.6 / 32.7 ms on two baseline builds, 33.7 ms streamed), while the 5.3-10.6 bits-per-pixel images stay 10-17% faster. - a tile worker thread the system refuses to spawn leaves fewer workers instead of panicking. Byte checks over 292 benchmark rows and 62 workload decodes (the 5661 x 8287 full decode included) pass: output identical to that of the planes route and to the reference hashes.
…qualify Two refinements of the default route for tiled images, measured on an 18-core ARM64 machine (warm / cold ms, footprint MB; bytes identical to the planes route and to the reference hashes everywhere): - a full-resolution tiled classic RGB image whose tiles do not decode at once (too few tiles for the workers) now streams them one after another when each tile meets the untiled rule as an image of its own (a megapixel or more, 4 coded bits per pixel or more, 2 to 11 pool threads): 4096 x 4096 RGB in 16 tiles of 1024 x 1024 on 6 threads 116.9 vs 124.9 ms on the planes (cold 118.4 vs 146.6), 70 vs 315 MB; smaller tiles keep the planes (one after another was 40% slower on 512 x 512 HTJ2K tiles). - reduced-resolution tiled images keep the planes: one tile after another was 1-13% slower on 6 threads and 1.5-10.6 times on 18, and tiles at once were never measured reduced (the default used them). Tiles at once stay wherever every worker gets a busy tile, measured with the caller + T scoped workers: 4096 x 8192 RGB in 32 tiles 219.1 vs 237.9 ms on the planes with 6 threads and 103.5 vs 134.8 with 18 (cold 218.0 vs 253.7 and 103.6 vs 152.6), 158 vs 564 MB; 4096 x 2048 HTJ2K in 32 tiles 7.59 vs 9.19 ms (6) and 4.61 vs 5.25 ms (18). Every full-image benchmark row is a one-tile image: their routes do not change.
The default route treated every reduced untiled classic decode alike and kept every full-resolution image under a megapixel on the planes. Measured on small images (an 18-core ARM64 machine, warm ms ratio streamed / planes, bytes identical), both are wrong in places: - small reduced classic decodes on 12 pool threads or more run on the pool's bands as slowly as on one thread: a 384 x 256 output of the 768 x 512 lossless images 4.86 times as long as on the planes on 18 threads, 512 x 512 images at 1/2 3.6-3.8 times; on 6 threads they are 1-12% faster. Outputs from 1024 x 512 up are 6-53% faster on 18 threads. Small HTJ2K outputs are up to 59% slower streamed. Reduced decodes now stream from 2^19 output samples as before, and below it only classic ones from 256 x 256 on 2 to 11 pool threads (256 x 256 and 384 x 256 outputs 3-12% faster on 6 threads, a 192 x 128 one 1-3% slower). - full-resolution classic images of 4 coded bits per pixel or more stream faster below a megapixel too: 768 x 512 (9.6 bpp) 0.90 on 6 threads and 0.91 on 18, 512 x 512 (6.6 bpp) 0.95 and 0.86-0.97, while a 4096 x 1024 one is 6-7% slower on 18. They now stream from 512 x 512 on 2 to 11 pool threads, and below 2^19 samples on larger pools. Tiles decoded one after another still need a megapixel each (measured on 1024 x 1024 tiles only). Full-image benchmark rows (4096 x 2048 and larger, outputs of 1024 x 512 and more) keep their routes.
…ged pool bands Re-picks the default routes now that classic pool bands decode in parts and tiles with deep band rings hold half a row batch, streaming wherever the streamed route was at least as fast as the planes (an 18-core ARM64 machine, one interleaved run, warm ratio streamed / planes, cold in brackets; bytes identical and equal to the reference hashes): - untiled gray classic images: 2896 x 2896 / 4096 x 4096 / 5792 x 5792 on 6 threads 0.88 / 0.87 / 0.85 (0.80 / 0.80 / 0.78), on 1 thread 1.00; on 18 threads and the default pool 5792 x 5792 is 1.00 (0.76- 0.79) while the smaller ones average 4-5% slower. They now stream from 2896 x 2896 below 12 pool threads and from 5792 x 5792 above; 5792 x 5792 footprint 459 -> 68 MB. - one thread: untiled classic RGB of a megapixel or more at 4 bits per pixel or more (0.99-1.00, 144 -> 46 MB) and tiled RGB images, whose tiles stream one after another (4096 x 4096 in 1024 x 1024 tiles 1.00, 262 -> 63 MB; HTJ2K in 512 x 512 tiles 0.93-0.99, 136 -> 38 MB). HTJ2K full-resolution images stay on the planes (streamed 1.04-1.45x), as do the rows listed in the change log.
… streamed rows A region decode that hands its rows to a sink holds no region-sized f32 planes. Image::decode_native_region_components_rows_with_context (hidden) runs that decode and packs each finished rectangle into the returned native-bit-depth planes with the planes route's own conversions (round_clamped_native up to 16 bits, push_native_sample_bytes above), so the planes hold the same bytes. Image::streamed_rows_native_precision (hidden) says when the rows carry the samples the native decodes pack: no palette, channel definition, sYCC or CIELab, unsubsampled components of one precision up to 24 bits. The planes are the decode's output and, like a caller's buffer, are not charged to its budget; each keeps the per-allocation cap.
…nt planes A 16-bit region decode of an image whose rows carry its native samples (no palette, channel definition, sYCC or CIELab; unsubsampled components of one precision up to 16 bits; an RGB or gray layout) now streams its rows through a U16RowSink instead of holding the region's f32 planes and a RawBitmap of its native samples. The sink converts each finished rectangle as write_u16_output converts the packed native samples. Other images keep the RawBitmap path, with its output and errors. The 16-bit component-plane writers convert a plane at a time (the per-plane choices made once, a 256-entry table for samples of up to 8 bits instead of a u32 division per sample, 8-bit samples as byte * 257, RGB one pixel at a time), with the per-sample conversion kept in the tests as the reference. J2kDecoder::decode_native_components runs as a region decode of the whole image that packs its rows into the native planes when the planes route would be refused, by its decode (the native planes floor) or by its pack, which holds every f32 plane besides the packed samples. decode_native_region_components streams whenever the rows carry the native samples, as 8- and 16-bit region decodes do. Tests: forced streamed routes write the planes route's bytes for 4/8/12/16-bit, signed and unsigned, 5/3 and 9/7, tiled, HT and RGBA images, and region decodes equal crops of the full image.
…med region The plan-time router (full_image_route) also streams Rgb16/Rgba16/Gray16 full-image decodes (decode_into, decode_scaled_into and the decoders behind decode_tiles_into) of images whose rows carry their native samples, when the planes route would be refused: by its decode (the native planes floor, as at 8 bits) or by its pack, which holds every f32 plane besides the RawBitmap of native samples (u16_rows_pack_floor, a lower bound of that charge). The streamed route's 16-bit arm is the U16RowSink region decode, so the output is the planes route's. Other 16-bit decodes keep the planes and their errors. The router edit is the 16-bit arm of the format match and the floor.max(pack) comparison.
The 16-bit component-plane writers walk the output in 2-, 6- or 8-byte pixels chosen once per row, so the per-sample conversion and interleave compile to fixed-size stores.
… does not fit A component decoding into planes whose band route planned within the budget, but whose rings, executor records or threads' scratch then did not fit beside the plan, failed with the budget's error even where the serial bands fit: the error reached the whole-window replay, which reports the original error when the replay succeeds. The executor now declines before anything decodes, and the component decodes in serial bands, as when its route declines while planning. The budget test now finds the least room a single component's route plans in, which leaves none for its executor, and requires the component to decode there in serial bands, bit for bit as the pool decodes it with room. A new test measures the whole-window tile route under the counting allocator, entered from a thread outside the pool that never entered it before: what the route holds stays within what it charges, its shared locks and the pool's entry included.
…ker count With an explicit worker count, each streamed image of a multi-image group decoded serially on its worker thread, and the static chunks of the batch plan left workers idle: 8 images on 6 workers ran on 4 threads, 8 on 18 on 8, and the last images ran alone. Such a group now decodes every image inside a rayon pool of exactly that many threads, kept by the `CpuBatchDecoder`, with automatic inner parallelism: idle pool threads take Tier-1 bands of the images still in flight. The worker threads only wait for their images and keep their admission claims, so the images in flight stay one per worker and the group never runs more threads than the count allows. `rayon` becomes a normal dependency of j2k; j2k-native's default `parallel` feature already links it. On an 18-core ARM64 machine, 8 copies of a 2048 x 2048 RGB image (NHWC) decode in 395 ms instead of 555 ms on 6 workers and in 192 ms instead of 301 ms on 18, and four half-scale decodes of a large RGB JP2 scan in 395 ms instead of 1057 ms on 18 workers. Eight workers for eight images were already busy (268 ms either way). Peak heap grows by the pool's band working sets (192 instead of 116 MiB for the RGB group on 6 workers), still far below the staged route's 1407 MiB. Output is unchanged.
With one explicit worker (up to four for Gray8), a classic group of several Gray8/RGB8 images whose plans each fit the per-decode bound, and together the batch execution cap, decodes them on the staged route again instead of streaming; larger groups and HTJ2K groups stream. The staged route decodes each code-block of a window's images back to back, which speeds up classic Tier-1 when the images repeat, and at these counts it keeps its workers busy, so the stream pool has no idle thread to gain from. Staged windows claim such an image's plan bytes, as they claim an offset plan's. On an 18-core ARM64 machine, 8 copies of one classic image streamed in 1.13-1.32 times the staged time at one worker, and 8 Gray8 copies in 1.09-1.24 times at two to four workers; those settings now measure 0.97-1.02 of the staged route (0.99-1.01 at one worker on a 10-core ARM64 machine). Distinct images streamed 4-5% faster at one worker, which the staged route gives up. RGB8 streams 10-40% faster from two workers on and Gray8 from six, and HTJ2K images stream about 20% faster even when they repeat, so those keep streaming. The plans' working memory returns for the staged settings (950-4302 MiB peak heap for the 8-image groups instead of 66-303 MiB). Output is unchanged.
Batch preparation labelled every Gray8/RGB8 image whose plan passed the streaming threshold or the per-decode bound as streamed, from its output format alone. The region decode streams rows only for images without a palette, channel definition, sYCC or CIELab, with unsubsampled components of at most 24 bits, and without image or tile offsets; any other image decodes whole component planes and hands them to the row sink at the end. An RGB JP2 with identity channel definitions, which batch accepts, was therefore admitted with the streamed claim while it held whole-image planes. Preparation now asks `Image::region_decode_streams_rows` (doc-hidden), which collects those header-level conditions, and applies the streaming threshold and the lone-Gray8 rule only to images for which it holds; the rest keep their offset plan, or are rejected up front when it exceeds the bound, like the other outputs. The test fixtures cover an RGB JP2 with a cdef box and RGB8/Gray8 codestreams with offsets: they decode the same samples as the plain images, keep their plans past the threshold, and are rejected under the bound while the plain images stream.
`decode_cpu_group` flattened a group's code-block payloads before it checked whether the group's images decode serially, which the staged route needs. A group whose images decode with inner parallelism, such as any lone image or a group under automatic workers on a host with more cores than images, discarded that work. The parallelism check now runs first. Two tests that counted the discarded flattening of lone images now expect none, and the HTJ2K pass-bucket test sets one worker so its group takes the staged route that decodes the flattened jobs.
With one explicit worker (up to four for Gray8), a classic group whose images may stream decoded all of its plans on the staged route whenever they fit the batch execution cap. The staged route only gains where identical code-blocks decode back to back, and it holds each staged image's whole-image plan scratch; distinct images stream faster at every worker count and hold a band of rows instead of whole-image planes. Every image whose region decode streams its rows now streams at every worker count, repeated or not. Groups of plans that do not stream keep the staged route at one worker. Their flattened payloads are now checked against the flattening and staged-window caps first: a group beyond them decodes its plans one image at a time instead of failing the batch. A planned image among streamed ones (`b` in `[a, b, a]`: an RGB8 JP2 with identity channel definitions, or a Gray8 image with offsets) took the group's policy only in the parallelism handed to the decode call. Its plan reads the policy from its worker, which the scheduler had set for a single job, so it executed with nested parallelism: on the global pool at one worker, and across the stream pool at two. The policy is now set on the worker before the decode, so images decoded one at a time run serially at one worker and in the explicit pool otherwise. The tests pin the route: copies, distinct and mixed groups stream at one to eight workers and with automatic workers, on a pool of the worker count, while the same images as plans stage at one worker; plans beyond the flattening cap decode one at a time with the same samples; and `[a, b, a]` with a planned `b` decodes it without nested parallelism at one and two workers.
…arly as fast On one thread (CpuDecodeParallelism::Serial or a one-thread pool) the default full-image route kept the component planes, about 12 bytes per pixel of working memory (more for reduced decodes, which hold the full-resolution coefficients), for several kinds of image the streamed route decodes as fast or within 3%. Measured on an 18-core ARM64 machine at one thread (7 interleaved rounds, one process per route and round, median of the per-round warm ratios streamed / planes; an A/A copy of the planes build stays within 0.987-1.007; bytes identical on every image), these now stream: - classic RGB below 4 coded bits per pixel from 4096 x 2048 samples, as on 2 to 6 threads: 4096 x 2048 to 5428 x 4460 images at 1.5 to 2 bpp 1.016-1.029 (cold 0.97-0.99); a 5428 x 4460 image 309.4 -> 16.0 MiB heap. 2048 x 2048 to 2896 x 2896 ones (1.035-1.041) keep the planes. - classic RGB at 4 bits per pixel or more from 512 x 512 instead of 1024 x 1024, as on 2 to 11 threads: 1.008-1.027, a fifth to a half of the heap (16 tiles of 512 x 512 decoded one by one 1.024). - tiled classic images whose tiles are 256 x 256 or more (512 x 512 before; HTJ2K tiles keep that floor): 1.008-1.014, 193 -> 0.9 MiB for 4096 x 4096 in 256 x 256 tiles. - reduced classic outputs from 512 x 256 (1024 x 512 before): 0.89-0.98, e.g. a 1/8 decode of 5428 x 4460 101.9 -> 1.6 MiB; and reduced tiled images: 0.96-0.99, 53 -> 1.2 MiB. - reduced HTJ2K decodes at 1/4 and below: 0.87-0.94 (cold 0.56-0.64), 41-83 -> 2 MiB. 1/2 decodes (1.009-1.093) keep the planes. - gray classic images from 1024 x 1024 (2896 x 2896 before): 1.004-1.026; reduced gray from 512 x 256: 0.97-1.00, 204 -> 3 MiB; tiled gray in tiles of 256 x 256 or more: 1.006-1.025. - 16-bit outputs of 512 x 512 samples or more, any coding, untiled or in tiles of 256 x 256 or more: the planes route packs them from planes of native samples it fills first, and streaming is faster: HTJ2K 0.755- 0.791, classic 1.5-2 bpp 0.913-0.941 (488 -> 16 MiB for 5428 x 4460), 0.987-1.011 elsewhere. Full-resolution 8-bit HTJ2K images (1.047-1.283 streamed, the planes' fused last level against the streamed line IDWT), HTJ2K gray images (1.048-1.147), smaller images and multi-thread routes are unchanged. The route request carries the decode's scale instead of a reduced flag. The router checks that the planes fit the decode budget, not that the streamed route's own buffers do: a very wide and shallow image (an 8,388,608 x 1 or x 2 RGB image of one level) fits its planes while its reconstruction rows pass the budget, so the stream is refused before it decodes. Such a decode now runs again on the planes, from a fresh context, when the stream was chosen by preference and the planes fit the default budget. This also covers the multi-thread preferences that predate this change (2 to 6 and 12 or more pool threads failed on that image before; one thread kept the planes).
decode_rows_u8_bounded and decode_rows_u16_bounded decoded every stripe as a region decode of its own. They share the parsed tiles, but no decoded code-block samples or reconstructed wavelet rows, so each stripe decoded again every code-block its wavelet support reaches and reconstructed the support rows around its edges: at the default 64-row stripes Tier-1 decoded 6.0 to 7.4 times the code-block samples of one full decode on 2048- to 8287-row images (2.5 to 2.9 times at 256 rows), and the horizontal inverse transform 2.2 to 2.3 times the samples. An image whose tiles span its width and whose headers let its region decode stream its rows now decodes once, as a region decode of the whole image whose rows are packed into a buffer of at most the requested stripe rows and handed to the sink a stripe at a time. A streamed decode that fails before any row reaches the sink decodes the image a stripe at a time instead, so such failures keep the rows and errors of the stripe-by-stripe decode. Rows, their bytes and order, the stripe bound and the errors are unchanged; a decode error after rows reached the sink leaves the stripes before it delivered. A sink that refuses a row still stops the rows at that row, but a one-pass decode now goes on until it completes or fails. The choice is planned from the main and tile headers: images whose tiles sit side by side, whose headers keep region decodes on component planes, or whose main header codes no decomposition level, and 16-bit rows of images deeper than 24 bits, decode a stripe at a time as before, and an image whose tiles turn out not to stream (a tile-level coding style without decomposition levels, HTJ2K code-blocks mixed with classic ones) decodes into component planes of the whole image, its rows handed over a stripe at a time. The stripe bound covers the packed output stripe, not the decoder's working memory. On an 18-core ARM64 machine (median of 7 interleaved rounds, 64-row stripes, time / peak heap / peak resident memory beside input and output, before -> after): - 5661 x 8287 RGB: 1 thread 12.9 s -> 1.34 s, 10.3 -> 22.5 MiB, 31.8 -> 25.6 MiB; 6 threads 2.54 -> 0.23 s, 12.0 -> 46.1 MiB, 76.8 -> 48.8 MiB; 18 threads 1.52 -> 0.11 s, 12.8 -> 69.0 MiB, 77.4 -> 72.8 MiB. - 5428 x 4460 and 6061 x 4252 RGB with selective bypass: 0.09 to 0.13 of the time at 1, 6 and 18 threads, heap up to that of their streamed full-image decode, resident memory down 8 to 34 MiB. - 4096 x 2048 HTJ2K: 0.13 to 0.21 of the time; on 6 and 18 threads heap 8-9 -> 72-73 MiB and resident memory 41-43 -> 76-77 MiB. - 256-row stripes: 0.21 to 0.51 of the time. - 8192 x 8192 in 1024 x 1024 tiles (stripe at a time): unchanged.
…mponent without deep rings A classic tile of several components that streams its rows from pool bands with a reader per component and full-width reads holds three bands in each finest-level ring and two in the others, plus one at every level with the deep rings of 12 threads and more at full resolution. Without the deep rings, as on 10 and 11 threads or at a reduced resolution, a three-component tile leaves 18 finest-level bands ahead of its readers. That is at least the pool's threads, so plan_parts splits no band, and a thread that finds no whole band to start waits. Such tiles now hold two bands in each finest-level ring when the pool has more threads than the tile has finest-level sub-bands. The nine bands ahead are then fewer than the threads, so they split into parts that every thread takes. Deep rings, one-reader tiles, HTJ2K tiles, single-component tiles and narrower regions keep their rings: on 6 and 8 threads two finest-level bands drain the pool between band rows (25-40% slower), and a single-component tile on 18 threads leaves too few bands to split (27% slower). Peak heap (counting allocator), output identical: - 5661 x 8287 RGB on 10 threads: 55.3 -> 49.2 MiB; at half resolution on 12 threads 32.2 -> 29.1 MiB, at a quarter 17.8 -> 16.3 MiB. - 18000 x 12000 RGB on 10 threads: 168.0 -> 148.4 MiB; at half resolution on 10 and 12 threads 89.1 -> 79.2 MiB. - 6061 x 4252 RGB on 10 threads: 51.7 -> 45.1 MiB. - 16384 x 16384 RGB on 10 threads: 152.3 -> 134.4 MiB. Other decodes hold the same bytes as before. Decode times, 7 interleaved rounds against two builds of the base: on a 10-core machine at 10 threads and the default pool, the full-resolution RGB images take 0.5-4% less time (5661 x 8287: 1.5-3%; 16384 x 16384: 4%; 18000 x 12000: 0.5%) and the 18000 x 12000 one at half resolution 6.5% less; on an 18-core machine at 10 threads the full-resolution ones take 5-7% less. Decodes the rule leaves alone run the same code. The row-sink test decodes three-component tiles on 9 to 12 threads and four-component ones on 12 and 13, at full and half resolution, and checks the deepest ring each level holds (a test-only counter of the planned rings) and that the bands split into parts exactly where the finest rings are shallow. With three finest-level bands it fails on 10 threads, and without the thread condition on four components at 12 threads. The band executor's counting-allocator test gains a wider tile, which takes the shallow rings on 10 and 11 threads and splits its bands into parts by itself, and checks the executor allocates within its charge there too; leaving the slot records of such a tile out of the charge fails it.
A batch with an explicit worker count keeps up to that many images in flight on one stream pool of as many threads. Each image's pool bands hold the rings of a decode that has the pool to itself: three bands at the finest level and two at the others, plus one at every level with the deep rings of 12 threads and more. The extra depth keeps a lone decode's threads busy while its readers cross a band row; when the batch plans as many images at once as the pool has threads, each thread starts with an image of its own and an idle one takes the bands of other images still in flight instead. DecoderContext gains a doc-hidden set_shared_pool (one line in the rustdoc-hidden API snapshot). The batch sets it on the context of every chunk it runs on its stream pool when the workers it plans at once are at least the pool's threads (shares_stream_pool): eight images on 2, 3, 4 or 8 workers, three on 2, but not eight on 6 (four at a time) or two on 18. An image encoded in more than one and a half times the group's mean bytes keeps the usual rings (shared_ring_bytes): Tier-1 work follows the coded bytes, so it is the likeliest to decode last with the pool to itself, where the shallow rings cost a lone decode 25-40%. The tile context carries the mark; from_workspace and into_workspace clear it, so a recovered workspace decodes with the usual rings. A decode with the mark on a pool of fewer than 12 threads holds two bands in every ring of its pool bands (shared_rings) and keeps its bands whole; the row batches are unchanged. Two-band rings leave fewer bands ahead than a 10-thread pool has threads, and splitting them into parts for the pool's other threads, which the other images keep busy, made ten images on 10 workers 1.3-1.6% slower on both machines. From 12 threads a lone tile holds the deep rings, and two bands a ring in their place made eighteen images on 18 workers 1.6-1.9% slower, with whole bands too, so the mark leaves those decodes alone. One band at the coarser levels saved more but made 4-worker batches 3-6% slower on a 10-core machine, as a reader then decodes those bands alone. Peak heap of the batch's first iteration (counting allocator, including the decoded output), output identical: - eight 2048 x 2048 RGB images on 2 / 4 / 8 workers: 121.4 / 147.3 / 200.6 -> 116.9 / 138.3 / 182.6 MiB; - ten 2048 x 2048 RGB images on 10 workers: 252.0 -> 229.5 MiB; - eight 2896 x 2896 gray images on 8 workers: 117.5 -> 108.9 MiB; - eight 2048 x 2048 HTJ2K RGB images on 4 workers: 239.8 -> 230.8 MiB; - eighteen on 18 workers: unchanged (736.3 MiB). Batch times, 7 interleaved rounds against two builds of the base on an 18-core and a 10-core machine, stay within the 1% noise on every one of these batches (ten on 10 workers 1.001 / 0.997, eighteen on 18 1.002). One noise image with 3.7 times the coded bytes among crops of the same shape, eight on 8 workers, took 2.2-2.5% longer with every image marked and takes the same time with the size cutoff (1.000, 0.999). Single-image decodes are unchanged. The row-sink test decodes a three-component tile on 4, 10 and 12 threads with and without the mark and checks the deepest ring of each level and the bands in parts: two bands a ring with the mark below 12 threads, and whole bands where on 10 threads a lone decode's two-band rings split them; on 12 threads the marked decode holds the lone decode's deep rings. The rows match the serial planes and a marked decode with fewer bands a ring peaks lower. A context recovered from the workspace of a marked decode holds the unmarked rings. Batch tests check which worker and image counts share the pool and which images the size cutoff leaves out. In the band executor's counting-allocator test, every case also decodes with the mark: marked decodes hold two bands in every ring below 12 threads and never the deep rings, within their charge.
… charged for A tile streaming from pool bands through a reader per component sizes its batch of rows by the output region's width, while the ROI plan charges the batch by the tile's final window and the buffer holds rows as wide as the tile. One rule sizes both, and it halves a classic tile's single batch only while that leaves 128 rows, so the two disagree when the region and the tile fall on either side of 8,192 samples: in an 8,200-wide image of 8,192-wide tiles the first tile was charged 128 rows of each component and held 255, twice the charge (6.4 M against 3.4 M samples), past the budget the decode had checked. The batch keeps the rows the region's width gives, capped by the rows the tile's width is charged for. Where a tile held no more than its charge nothing changes; the case above now holds 128 rows. Output is unchanged. The new test decodes that image on 12 threads, a reader per component, and checks the batch's samples and that the line buffer stays within its charge; it fails without this change. On an 18-core ARM64 machine, 7 interleaved rounds, an 8200 x 2048 image in 8192-wide tiles decodes in 0.979 of the time on 12 threads and 0.983 on 18 (a second copy of the old build: 0.997 and 1.001), peak heap 72.6 -> 60.7 MiB; 8192 x 2048 in 4096-wide tiles and 4096 x 4096 in 1024-wide ones are unchanged (1.003 / 1.001 and 0.999 / 1.002, same heap), as on a 10-core machine on 10 threads.
…1 workspace An HTJ2K code-block that decodes straight to samples writes every row and column of the block, so one that the sub-band stores only part of decodes into scratch first. The serial sub-band decode allocated that scratch on first use, against the budget. A streamed region decode reaches the blocks its region clips at the bottom only after its first rows went to the sink, so with a budget that fit everything else it failed there, with a capacity error, after rows went out. The retries that rerun only a decode that has handed nothing over (without the retained scratch, or without the packet directory) were then closed to it. The serial Tier-1 workspace now plans the largest clipped block's scratch with the rest of the workspace and sizes it before the first band; it is kept beside the HTJ2K context and counted with it. A decode short of budget now fails before its first row. Output is unchanged. A tile with clipped HTJ2K blocks holds one more block of samples (at most 16 KiB) through its decode. The new test decodes classic and HTJ2K regions, two of them clipped at the bottom, from budgets at and above the largest one that fits: each either decodes every sample once or fails with a capacity error before its first row. It fails without this change.
The row-streamed IDWT ran every vertical lifting step on one row at a time, each step its own pass over the row and its two neighbours. The banded IDWT and the planes route's fused last level run all the steps of a row's arrival in one sweep with the rows in registers (`lift_arrivals`). The streamed IDWT now lifts every level through that kernel, one arrival at a time, on the ring of rows each level already holds (a row per step and two more), so its memory is unchanged and both routes share one lifting kernel; `lift_one_row` and the per-step schedule go. A level hands a row over once its last step has run, which is when the per-step schedule finished it, so a decode reads the same band rows before each row it returns as before. Output is bit-identical. On an 18-core ARM64 machine, timers in the decode, a 5428 x 4460 RGB image at 0.5 coded bits per sample on 1 thread: last-level lifting 13.8 -> 8.9 ms, coarser levels 9.2 -> 7.9 ms; an 18000 x 12000 HTJ2K image: 151 -> 106 and 81 -> 70 ms.
A streamed decode on serial bands copied each finished row of every component into a scratch row, transformed the copy in place (the inverse MCT, the level shift and rounding) and packed it or handed it to the sink; only 9/7 rows with the MCT and an RGB8 destination skipped the copy through the fused ICT pack. The IDWT can't hand out its own rows for an in-place transform as soon as they are final, because the next row's last lifting step still reads them. `LineReconstruction::next_row_mut` hands a row over once no lifting step reads it any more, one arrival later than `next_row`, which the level's ring already holds, and `RowOutput::write` transforms and packs each row there: no scratch rows (their buffer and charge go) and one sweep less for every transform. The per-row RGB8 finish of three components with the MCT (`DeferredMct::pack_rgb8_row`: the fused ICT pack where it exists, else the transform and the standard pack) is now shared by the planes' fused last level, the serial rows and the pool's row batches. Output is bit-identical; rows reach the sink one arrival later than before (the last one, which nothing reconstructs after, as before), which only changes how many rows a decode that fails partway leaves written. Each row of every component is taken final (`finish_next_row`) before any takes the arrival that frees it, so the components ask for their band rows, and decode their bands, in the order they did: a tile with failing code-blocks meets the same one first and reports the same error. On a 12-core ARM64 machine, 7 interleaved rounds, against the planes route of the same build: a 4096 x 2048 lossless HTJ2K image on 1 thread 1.020 -> 0.972 of the planes' time.
A tile streaming from pool bands through one reader (classic tiles below the reader-per-component rule, and one-component tiles) reconstructed 32 rows of every component into a row batch, copying each finished row out of the IDWT's ring, then transformed and packed the batch on the same thread: a copy and a buffer that overlap nothing. That reader now goes through `RowOutput::write`, finishing each row in place in the ring as serial bands do; it still decodes bands itself while it waits. The batch, its charge and the batch path's one-reader branch go: only a reader per component, whose rows meet on the pool, holds row batches. Rows reach the sink one at a time rather than 32 at a time, which only changes how many rows a decode that fails partway leaves written. Tests inject a failing band at many points on serial bands, one reader and a reader per component, into an RGB8 destination and as rows: every row handed over is the clean decode's, once, from the top. A band stranded after rows were handed over ends the decode with an error, before them the tile still decodes on serial bands; and a decode over its budget, of the whole image or of regions clipped at the bottom, fails before its first row, so the retries without retained scratch or without the packet directory still run and succeed. Output is bit-identical. On a 12-core ARM64 machine, 7 interleaved rounds, against the planes route of the same build, 6 threads: a 5428 x 4460 RGB image at 0.5 coded bits per sample 1.036 -> 1.018 of the planes' time, a 6061 x 4252 one 0.965 -> 0.929; peak heap 36.7 -> 34.7 and 41.7 -> 39.4 MiB.
… fails A tile whose components reconstruct on readers of their own, two batches of rows at a time, packs one batch into the sink's RGB8 destination while the next reconstructs. When that reconstruction failed, the error returned before the packed batch was counted as streamed, though its rows were already in the destination. A band stranded there (a scheduler fault) then found nothing handed over and decoded the tile again on serial bands, writing those rows a second time; any other error left the count short. Packed rows now count as soon as they are packed, whatever failed alongside, so a strand after them ends the decode with an error, as for rows handed to `write_rows`. Errors keep their precedence and output is unchanged. The publication tests' destination now counts every row written into it, blanking what it has seen each time the decoder asks for it, so a row written twice shows even with the same bytes; the stranded-band test fails without this change.
An HTJ2K tile streaming from pool bands through a reader per component holds two batches of every component's rows, each as many rows as 2 Mi samples fit, up to 512: on a 4096-wide RGB tile, 2 x 512 rows of 3 components, 48 of its 72 MiB peak. Two batches now keep to 256 rows each (`STREAMED_PIPELINED_MAX_ROWS`), the smallest size measured that keeps the batches' pipeline at speed. A tile holding one batch (classic) and tiles whose batches are shorter already, such as 18000-wide ones (116 rows) or 1024 x 1024 regions (256), are unchanged. Output is bit-identical. A 4096 x 2048 HTJ2K image at 2 bits per pixel, 7 interleaved rounds, against 512-row batches of the same build (a second copy of that build in brackets): on an 18-core ARM64 machine 1.005 [0.997] of the time on 12 threads, 0.986 [1.005] on 18, 0.983 [0.985] on all cores (11 rounds; a lossless image of that size 1.012 [0.998]); on a 10-core one 0.994 [0.993] on 10 threads. Peak heap 71.8 -> 47.8 MiB; the first decode takes 0.87-0.91 of the time. 128-row batches would hold 36 MiB but take 3-10% longer, and 6-15% on 1024 x 1024 regions: each batch ends at a barrier where the readers wait for the slowest component, and smaller batches meet it more often.
Share record sizing, tile bounds, streamed line planning, classic row reconstruction, HT scratch preparation, strip execution and typed row-sink plumbing. Avoid flat VLC reservation when SIMD is disabled while preserving the specialized kernels and public APIs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large full-image decodes can exceed the working-memory budget when they allocate whole component planes. This change selects the existing streamed row pipeline at plan time for eligible full-image, reduced, region and batch requests, including 16-bit, signed and native-component output. It also makes bounded-row decoding use one streamed pass instead of reconstructing each stripe separately.
The implementation builds on the banded Tier-1 and line-IDWT work in #133 and #310. It adds reconstruction across component readers, splits bands into worker-sized parts where needed, gathers multi-segment payloads per block, and accounts for reader batches, rings and worker scratch before execution. Shared planning, IDWT dispatch, conversion and sink helpers replace duplicated paths. Fixes cover empty coarse levels, clipped-block scratch and conversion of samples deeper than 16 bits.
For a 5661×8287 RGB image emitted in 64-row stripes, seven interleaved rounds on an 18-core ARM64 machine measured 12.9 s → 1.34 s at one thread and 1.52 s → 0.11 s at 18 threads, with identical output. One-pass rows use the streamed full-image working set: the one-thread peak heap in that comparison rose from 10.3 to 22.5 MiB. The packed stripe limit bounds delivered output, not the decoder’s total working memory.
The supported decode signatures stay unchanged and route selection remains automatic. The implementation API snapshot records ten doc-hidden helpers, including an isolated test hook for the planes budget; the API evidence and semver checks are updated together.
Relationship to #133
Changed files with prior jcoyne contributions (45)
Validation
Focused native default/std-only, output and streaming tests pass, along with formatting, strict and upstream clippy, no-std, stable API, semver and repository lint. Existing pixel and allocation comparisons cover classic/HT, reduced/region, native/16-bit output and owned/staged batches; cleanup controls found no reproducible slowdown.
The full macOS suite also passes on the combined stack tip: 5,656 passed, 0 failed, 52 ignored. This is integration evidence. Separate exact-head CUDA and Metal quick validation passed on this PR’s exact head.