Repository navigation
perf(j2k): improve CPU decode kernels and prepared batch execution - #314
Merged
Merged
Conversation
…t once When PLT lengths cover the tile, a packet of an unused precinct is skipped by its length. In position-major progressions the precinct's remaining layers come right after it, so skip them in the same step (up to the end of the tile-part) and advance the progression past them, instead of walking each packet through the progression and the relevance checks. TilePartCursor::skip_packet, unused once skip_packets exists, is removed. Measured with the previous commit (progression merge); see there.
read_bits_with_stuffing takes each byte's remaining bits at once instead of one bit per iteration; reading the last bit of a 0xFF byte still consumes the stuff bit that starts the next byte (B.10.1), so values and positions match bit-by-bit reads (randomized test). The coding-pass count probes the header once instead of up to seven separate peeks. Measured against the previous commit, on an 18-core ARM64 machine, 5 interleaved rounds of J2K_PROFILE_STAGES (packet-header parse of a region of the no-PLT test images): the middle region of the 18000 x 12000 no-PLT image -22% on 18 threads and -20% on one; that of the HTJ2K image -30% on 18.
…' geometry ResolutionTile::precincts() becomes a validated PrecinctGrid with closed-form positions (the old iterated next_multiple_of) that can also answer one precinct's rectangle by index. Allocation planning, build_precincts and the region precinct selection loop precinct indices and compute a rectangle only for precincts they use; unused ones get no geometry at all. Selection hoists the per-resolution sub-band windows out of its precinct loop. Measured against the previous commit, on an 18-core ARM64 machine, 5 interleaved rounds: on the 18000 x 12000 test images: the HTJ2K one reduced four levels -7% on 18 threads; regions of the PLT one -3.7% (top-left), -3% (middle) and -2.8% (bottom-right) on 18 threads, -2% (middle) on 6.
A packet whose precincts are parsed only to reach later packets borrows a scratch precinct whose code-blocks have no layer records, so its header skips every contribution. Skip its body as one run of the summed skipped bytes (PacketSkips keeps a saturating total) instead of walking every code-block of its sub-bands; a short body fails with the same error. Measured with the next commit (record lookup) against the commit before, on an 18-core ARM64 machine, 5 interleaved rounds: regions of the 18000 x 12000 no-PLT test image -4.7% (middle) and -3.7% (top-left) on 18 threads; PLT rows within 1%.
record_offset binary-searches the run's precinct records for every packet. Position-major progressions read a precinct's layers in a row, so keep the last (component, resolution, precinct) and its offset; the selection is immutable while parsing, so the cached offset is the looked-up one. Measured with the previous commit; see there.
Read Table B.4's codewords prefix by prefix (1, 1, 2, 5 and 7 bits) instead of peeking up to nine bits and re-reading the matched prefix. Values and reader positions match the peek chain on success (randomized test). Measured with the next commit against the commit before, on an 18-core ARM64 machine, 7 interleaved rounds of J2K_PROFILE_STAGES: packet-header parse -5% to -8% on the regions of the 18000 x 12000 no-PLT and HTJ2K test images; 5 rounds of whole decodes of the no-PLT image's regions: middle -3.2% and top-left -3.1% on 18 threads, middle -2.1% on one.
…t walking its passes Without per-pass termination or selective bypass, every pass of a classic contribution belongs to the layer's one segment: push that segment directly (same index, pass count and length read) instead of grouping pass by pass. Measured with the previous commit; see there.
…e nodes ParseScratch::lend counted each tag tree's nodes, then TagTree::place counted them again; place now takes the count. The scratch code-blocks are filled row by row instead of dividing each index by the grid width. Measured against the commit before, on an 18-core ARM64 machine, 7 interleaved rounds of J2K_PROFILE_STAGES: packet-header parse -0.6% to -1.5% on the regions of the 18000 x 12000 no-PLT and HTJ2K test images (-1.1% HTJ2K).
The direct-output cleanup path, which decodes every cleanup-only block straight to dequantized samples, still read the VLC bits through the reverse reservoir reader. Its top-up checks the eight bytes below the cursor for stuffing on every quad pair, about a seventh of the pair loop's instructions. - The VLC bits are now unstuffed once per block, eight bytes at a time when none is stuffed, into a forward buffer in the order the reverse reader consumes them, then zero padded as that reader pads. It keeps the bytes a block can consume (at most 30 bits per quad pair), so the buffer is sized from the block's geometry: 1937 bytes for 64x64. - `FlatVlcReader` tops its reservoir up from that buffer with no checks and no branch, and hands each pair's first lookup the word from before the top-up: it already holds the seven bits that lookup needs, since a pair consumes at most 30 of the 56 a top-up leaves. - The pair decoders take either reader through `VlcBits`; the other HT paths keep the reverse reader. Tests compare the flat reader with the reverse reader on random stuffing-heavy segments and check the unstuffing limits. On a 10-core ARM64 machine, interleaved against the base and including the previous commit, the 4096x2048 RGB HT lossless decode is 9.8% faster at 1 thread, 7.1% at 4, 6.6% at 10 and 5.6% at the default; HT 2 bpp is 3.6% faster at 1 thread, 2.8% at 4, 1.8% at 10 and 2.6% at the default; 512x512 and 256x256 HT tiles 2.1% and 1.9% faster at 1 thread. 1024x1024 regions of an 18000x12000 HT image are 1.9 to 2.1% faster at 1 thread and 0.4 to 0.5% at 10; 8 concurrent region decodes are unchanged within noise. The HT decode workspace grows by the flat buffer. Output is unchanged.
PLT and PLM markers sized their length vector by decoding every length once, then decoded them again to store them. The decoding walk cannot fail in its arithmetic (a 7-bit shift into cleared bits), so its count is the number of bytes with a clear high bit, and its only error is a last byte that leaves a length unfinished. Count that way; the storing pass is unchanged (randomized test against the walk). Measured with the next commit (unread resolutions) against the commit before this one, on an 18-core ARM64 machine, 5 interleaved rounds, on 18 threads: the 18000 x 12000 PLT test image's middle, top-left and bottom-right regions -1.9%, -0.6% and -0.5%, the image reduced four levels -0.4%; the HTJ2K one reduced four levels -1.8%; other rows within noise.
…hout their geometry When the ROI plan reads none of a resolution's sub-bands (a reduced-resolution decode's finer levels), region selection marks its precincts unused without computing each precinct's rectangle and code-block grid. The index check the rectangle lookup made is kept: PrecinctGrid::rect_count is the number of indices rect_at answers. Measured with the previous commit (PLT length count); see there.
An Auto full-image decode whose estimated serial time (1 ns per component sample plus 1 ns per coded HT byte or 72 ns per classic byte) is below 25 us per pool thread runs Serial: Tier-1, IDWT, MCT and pack on the calling thread. Explicit parallelism and region decodes are unchanged, and a non-Auto request no longer asks rayon for its thread count (which would start the pool). Tests keep their parallel routes unless they opt in. Measured against the previous commit on an 18-core ARM64 machine, 5 interleaved rounds: HT lossless 128^2 -56% on 18 threads and -20% on 6; HT 2 bpp 256^2 -26% on 18; classic 256^2 +0.5% on 18, HT 512^2 +0.3% on 6; region rows (not affected) within +-1.6%.
The direct-output cleanup path folds each quad pair's bottom-row `v_n` into the next row's context. It extracted the four lanes to scalars, combined them with the previous pair's last lane, and stored two words plus a carry, about ten instructions per pair in a loop that is close to its instruction-throughput limit. The pair decoder now returns the bottom row as a vector. The fold slides the previous pair's bottom row in by one lane, ORs, and stores lanes 0 and 2 as one 64-bit word; the carry stays a vector until the end of the row. On a 10-core ARM64 machine, interleaved against the base and including the earlier commits, the 4096x2048 RGB HT lossless decode is 10.6% faster at 1 thread, 8.1% at 4, 6.1% at 10 and 6.4% at the default; HT 2 bpp is 3.7% faster at 1 thread, 3.2% at 4, 2.6% at 10 and 3.1% at the default; 512x512 and 256x256 HT tiles 2.7% and 2.3% faster at 1 thread. 1024x1024 regions of an 18000x12000 HT image are 2.2 to 2.6% faster at 1 thread and 0.1 to 0.3% at 10; 8 concurrent region decodes are unchanged within noise. Output is unchanged.
When a reversible code-block's passes decode down to bit-plane 0, so that Equation E-8 adds no midpoint, with no ROI shift and at most 31 coded bit-planes, every coefficient converts as decoded: a magnitude below 2^31 converts exactly through u32, and negation commutes with the rounding of the conversion and of the product, so the sign is a bit flip. Such rows now convert in a branch-free u32-to-f32 loop that vectorizes, on the tile route and on the serial per-sub-band path, instead of reconstructing each coefficient through i64. A unit test checks the row against the per-coefficient reconstruction for exact and inexact blocks. Classic lossless full-image decode, interleaved A/B against the parent: -5.6% at 1 thread, -4.6% at 6, -6.7% at 18, -5.0% at the default count. 2 bpp (irreversible) is unchanged. Output is byte-identical.
In the classic significance propagation and cleanup passes, a coefficient's sign is decoded right after the decision that makes it significant, a branch the data makes hard to predict. Its sign label (from the neighbors' flags) and context are now formed and loaded before the significance decision, so after a mispredicted significance the sign decode no longer waits for them. Sign labels (9-13) never equal zero-coding labels (0-8), so the significance decision cannot change the context loaded early. Classic full-image decode, interleaved A/B against the parent, for the significance propagation pass: lossless -7.3% at 1 thread, -7.1% at 6, -4.6% at 18; 2 bpp -3.0% at 1 thread, -3.1% at 6, -2.6% at 18. The cleanup pass adds another 1-2% at 1 and 6 threads (2 bpp -1.3% to -2.1%, lossless -0.8% to -1.2%) and is within noise at 18. Output is byte-identical.
…d runs instead of sorting RPCL and CPRL list each component's and resolution's precincts in raster order, which is already their progression order whenever the reference-grid positions increase. Interleave those runs per leading-key group (resolution for RPCL, component for CPRL) instead of sorting every precinct of the tile. A tile whose runs are not strictly ordered, and PCRL, sort as before; the merge of sorted runs under the same total order yields the same packets. Measured with the next commit (PLT run skip) against base, on an 18-core ARM64 machine, 5 interleaved rounds, median of per-round ratios: on the 18000 x 12000 test images: the HTJ2K one reduced four levels -7.7% on 18 threads; regions of the PLT one -3.5% (bottom-right), -1.9% (top-left), -1.4% (middle) on 18 threads and -2.1% (middle) on 6; the no-PLT one's top-left region -2.4% on 18; one-thread rows within 1%.
Code blocks with SigProp or MagRef passes decoded their cleanup pass in two sweeps: every quad row's VLC/MEL symbols, then a separate MagSgn pass over the stored symbols. Cleanup-only blocks already use the fused row decoder, where the latency-bound VLC chain of one pair overlaps the MagSgn work of the previous one. Refinement-bearing blocks now take the same fused phase; they keep the zeroed symbol scratch the sigma build reads past the block. The split MagSgn phase stays as the test reference. MagRef visited all 32 samples of each 8x4 group through nested column and row branches. It now walks the set bits of the group's significance word, whose bit order (4 * column + row) is the scan order the refinement bits follow. On a 10-core ARM64 machine, interleaved against the base and including the earlier commits, 512x512 and 256x256 HT tiles encoded at 2 bpp (most of their blocks carry refinement passes) decode 9.9% and 8.2% faster at 1 thread. The 4096x2048 RGB HT lossless decode stays 10.7% faster at 1 thread and HT 2 bpp 4.2%; 1024x1024 regions of an 18000x12000 HT image 2.0 to 2.1% at 1 thread; 8 concurrent region decodes unchanged within noise. Output is unchanged.
…C bits from a flat buffer Two changes to code blocks that carry refinement passes. SigProp decoded each 4x4 group one candidate sample at a time, with a branch on every refinement bit (a one makes the sample significant and spreads candidacy to its later neighbors). It now decodes a column of four samples per lookup: a 4096-entry table indexed by the column's candidates, its still-insignificant samples and the next four bits gives the newly significant samples and the bits used, and the new samples make their neighbors in the next column candidates. A test checks the column decode against the candidate scan on random groups. The fused cleanup/MagSgn rows these blocks now use read their VLC bits through the reverse reservoir reader, checking for stuffing on every pair. They now unstuff the VLC bits once per block into the flat buffer the direct-output rows use, and inline the MEL and VLC refills as those rows do. The scalar rows keep the reverse reader. On a 10-core ARM64 machine, interleaved against the base and including the earlier commits, 512x512 and 256x256 HT tiles encoded at 2 bpp decode 11.1% and 10.9% faster at 1 thread. The 4096x2048 RGB HT lossless decode stays 10.7% faster at 1 thread and HT 2 bpp 4.3%; 1024x1024 regions of an 18000x12000 HT image 2.1 to 2.5% at 1 thread; 8 concurrent region decodes unchanged within noise. Output is unchanged.
…rect rows The direct-output row walkers (pair loop with the MEL-run shortcut, flat VLC and MagSgn buffers, vector v_n fold) served only cleanup-only blocks; refinement-bearing blocks decoded their cleanup pass through the older fused row loop, about 12 to 21% slower on the same blocks. The direct rows are now generic over what they publish per sample: `FinalOutput` keeps the dequantized f32 conversion, `SignMagnitude` stores the sign-magnitude words that SigProp and MagRef refine. Blocks whose width is a multiple of four (and within the SIMD bit budget) take the direct rows unless the phases are timed apart; others keep the fused loop. Tests check the sign-magnitude rows against the scalar rows on encoded and arbitrary blocks. On a 10-core ARM64 machine, interleaved against the base and including the earlier commits, 512x512 and 256x256 HT tiles encoded at 2 bpp decode 14.8% and 13.9% faster at 1 thread. The 4096x2048 RGB HT lossless decode stays 10.7% faster at 1 thread and HT 2 bpp 4.3%; 1024x1024 regions of an 18000x12000 HT image 2.1 to 2.3% at 1 thread and unchanged at 10; 8 concurrent region decodes unchanged within noise. Output is unchanged.
On one thread the fused last level kept three rings of 21 full-width rows, about 1 MiB for a 4096-wide image, so its horizontal synthesis, vertical lifting and pack ran out of L2. Each batch now runs column strip by column strip: a strip's synthesis and lifting of all three components and the pack of its part of the finished rows, on rings of its own of 1024 columns, which stay in L1. - Lifting is per column, so strips are independent. A strip's horizontal 9-7I synthesis starts the register-window pass at its first block: `lift_row_97_window` now takes a first block and lifts the two blocks before it as far as that block's outputs need, with the same expressions, so the output is bit-identical. The last strip also runs the row's multi-pass tail, factored out of `synthesize_row` as `synthesize_97_tail`, before its window, as `synthesize_row` does. - Only wide even-origin 9-7I levels are split; other levels, and the parallel last level, keep full rows: on the pool the shorter row segments cost more than the strips save. - `scaled_blocks` replaces the block-load closure of the window pass, so it stays inside the dispatch's target features however often it is called. - Tests run 64-column strips, so the fused-last-level fixtures cover several strips, a remainder strip and the row tail. HT 2 bpp full-image decode (4096x2048 RGB, an 18-core ARM64 machine), on top of the previous commit: -3.2% at 1 thread; the pool paths are unchanged. Output is unchanged.
MagRef walked the set bits of each 8x4 group's significance word, one sample and one refinement bit at a time. In the refinement-bearing blocks of rate-controlled HT tiles about two thirds of a group's samples are significant, so the walk ran some twenty serial steps per group and ended on a hard-to-predict branch. A group that lies entirely inside the block is now refined one row of eight samples per vector: each significant sample's refinement bit sits at its rank among the group's significant samples in bit order (an exclusive prefix sum of the column counts plus the rows above it), and the update is masked to the significant samples. Groups on the block edge keep the per-sample walk. A test checks the phase against the per-sample scan on random blocks, strides and densities. On a 10-core ARM64 machine, interleaved against the base and including the earlier commits, 512x512 and 256x256 HT tiles encoded at 2 bpp decode 16.8% and 16.3% faster at 1 thread. The 4096x2048 RGB HT lossless decode stays 10.6% faster at 1 thread and HT 2 bpp 4.2%; 1024x1024 regions of an 18000x12000 HT image 2.2 to 2.4% at 1 thread and unchanged at 10; 8 concurrent region decodes unchanged within noise. Output is unchanged.
After a SigProp group's significance bits, its newly significant samples take one sign bit each, in bit order. The loop over the new samples ran two or three times per group and ended on a branch the predictor missed about once per group; it was the hottest point of the SigProp scan in rate-controlled HT tiles. A group that lies entirely inside the block now writes its new samples one row of four per vector: each new sample's sign bit sits at its rank among the group's new samples (an exclusive prefix sum of the column counts plus the rows above it), and the other samples keep their value. Groups on the block edge keep the per-sample loop. A test checks the phase against the original candidate scan and sign loop on random blocks, strides, densities and stripe-causal settings. On a 10-core ARM64 machine, interleaved against the base and including the earlier commits, 512x512 and 256x256 HT tiles encoded at 2 bpp decode 18.6% and 17.7% faster at 1 thread. The 4096x2048 RGB HT lossless decode stays 10.5% faster at 1 thread and HT 2 bpp 4.3%; 1024x1024 regions of an 18000x12000 HT image 2.3% at 1 thread and unchanged at 10; 8 concurrent region decodes unchanged within noise. Output is unchanged.
The 5-3R horizontal synthesis copied both band rows into ghosted scratch, lifted the low samples in one pass and the high samples with the interleave in another. An even-origin row now runs both steps a block apart in one pass of SIMD_WIDTH-pair blocks, with the neighbours taken across blocks by `slide`, as the 9-7I rows already do; the multi-pass steps keep the row's last pairs, from two pairs earlier, and the pass then overwrites their wrong left edge. Every sample sees the same expressions, so the output is bit-identical. The pass can start at any block, so the serial last level's column strips now cover 5/3 levels too. HT lossless full-image decode (4096x2048 RGB, an 18-core ARM64 machine): -4.4% at 1 thread, -6.3% at 6, -2.0% at 18, -2.5% at the default, against the previous commit. Output is unchanged.
…the tile route A classic tile with fewer rows of code blocks than pool threads fell back to the per-sub-band path, which decodes the sub-bands of a small tile one after another. The tile route now keeps such tiles: the row splitting from the parent commit already balances their few long rows across workers. The band route keeps its per-sub-band fallback. Measured against the parent (built twice, as A and an A/A control) on an 18-core ARM64 machine, 5 interleaved rounds, median of per-round ratios: lossless classic 128^2 at 18 threads -51%, classic 2 bpp 192^2 at 18 threads -18%; every other row (classic and lossless 128^2-768^2 at 1/6/18 threads, HT 128^2-768^2, classic and HT mosaics, 8 concurrent region decodes, PLT and no-PLT regions) within the A/A spread. Output is byte-identical. The heap peak of lossless 128^2 at 18 threads rises from 0.31 to 1.12 MB, the same as the parent at 16 threads, where the tile already took the route.
The cleanup pass decodes a quad pair through three dependent table lookups: the two VLC codewords, then the U-VLC pair. Between them the codeword length, the context of the next quad and the U-VLC index were each rebuilt from the table symbol, and the reverse VLC reader refilled whenever fewer than 32 bits were left, a branch taken about every other pair that resolves only after the U-VLC lookup. - The VLC tables are relaid out at compile time from the existing tables. An entry holds the codeword length alone in its low six bits, so the reservoir shifts by the entry directly, the context bits it passes to the next quad in place, the U-VLC offset flag at the bit the U-VLC index needs, and the original symbol above them. The original tables, and the adapter API that exposes them, are unchanged. - The pair loops top the reservoir up without that branch: the eight bytes below the cursor are ORed in above the buffered bits and the whole bytes that fit are consumed. A stuffed byte among them, or fewer than eight left, takes the existing byte loop. Bits left above the count are the next byte's own bits, so ORing them in again is exact. A test reads random segments, stuffing-heavy and past their end, through both refills and compares every pair-sized step. On a 10-core ARM64 machine, interleaved against the base, the 4096x2048 RGB HT lossless decode is 6.1% faster at 1 thread, 7.5% at 4, 3.4% at 10 and 4.2% at the default; HT 2 bpp is 0.8 to 1.4% faster; 512x512 and 256x256 HT tiles 1.2% and 1.1% faster at 1 thread. Region decodes of an 18000x12000 HT image and 8 concurrent region decodes are unchanged within noise. Output is unchanged.
…umn table The SigProp column table gave each column's new samples and bits used; the scan then derived the candidates they add to the next column with three shifts and ORs, and counted the group's sign bits with a popcount after its last column. Both sat on the serial chain from one column's lookup to the next and from one group's bits to the next group's. Each entry now also holds that spread, already in the place where the next column's index takes its candidates, and the count of new samples; the bits used sit in the entry's low bits so the entry itself is the shift past them. A column feeds the next through a mask, and the group's bit count is a sum of entry fields. The existing test checks the table against the candidate-by-candidate scan, now with the sign count. On a 10-core ARM64 machine, interleaved against the base and including the earlier commits, 512x512 and 256x256 HT tiles encoded at 2 bpp decode 20.1% and 19.6% faster at 1 thread (1.8% from this change). The 4096x2048 RGB HT lossless decode stays 10.5% faster at 1 thread and HT 2 bpp 4.2%; 1024x1024 regions of an 18000x12000 HT image 2.4 to 2.6% at 1 thread and unchanged at 10; 8 concurrent region decodes unchanged within noise. Output is unchanged.
…sweep The streamed IDWT ran each vertical lifting step of an arrival as its own sweep over a row and its two neighbours: three loads and a store per sample per step, four steps per row pair for 9-7I. Every interior arrival whose row above has the first step's parity runs all of its steps, one row further up each, so those steps now run together: - `lift_arrivals` drives the vertical steps of a batch of arrivals for the whole-level, parallel-strip and fused-last-level streams alike. Arrivals near a level's or strip's edges, where a neighbour is mirrored or outside the strip, keep the one-step-at-a-time path. - Interior wavefronts go to `lift_wavefronts`, which sweeps the columns once for two wavefronts and keeps the rows they share in registers, so each row is loaded and stored about once. Every column sees the same operations in the same order, so the output is bit-identical. - The streams synthesize 16 arrivals ahead of their vertical steps; the ring and halo capacity arguments are in the comments. HT 2 bpp full-image decode (4096x2048 RGB, an 18-core ARM64 machine): -4.5% at 1 thread, -5.0% at 6, -4.9% at 18, -3.0% at the default; HT 512x512 tiles -7.1%. Output is unchanged.
The rounding of 9/7 planes that are not inverse-transformed by the MCT (`SampleShift` round-then-add and the float sign shift) was a scalar `round_ties_even` loop, a `rintf` call per sample on x86-64. On x86-64 it now goes through the dispatched `round_centered_and_shift`; ARM64 keeps its single-pass scalar copy loop. Output is byte-identical. Measured on x86-64 (Zen 3, AVX2) on an HT 2 bpp 4096x2048 RGB image encoded without the colour transform, median of per-round ratios over 9 interleaved rounds against the build without this commit: -38.3% at 1 thread (-58.5% against the base build), -10.1% at 8 threads.
…ersion The magnitude is masked to 31 bits, so the signed int-to-float conversion is exact and gives the same value. On x86-64 AVX2 it is one instruction; the unsigned conversion is a five-instruction emulation. Neon converts either way natively. Output is byte-identical. Measured on x86-64 (Zen 3, AVX2), median of per-round ratios over 5 interleaved rounds against the previous commit: 18000x12000 HT overviews (reduce 2/3/4) -1.3% to -2.3% at 1 and 16 threads, HT 1024x1024 regions -0.9% to -2.4% in-process; full-image HT decodes and 512x512 tiles unchanged (A/A spread 0.4%).
…-64 SIMD dispatch On baseline x86-64 every MQ decision's `leading_zeros` is a BSR plus a zero fixup on the per-symbol dependency chain, and the renormalizing shifts go through CL. The packed-column passes now run inside the runtime dispatch on x86-64, with the passes inlined into it, so they compile with LZCNT and BMI2 where present. ARM64 is unchanged (CLZ is baseline there). Output is byte-identical. Measured on x86-64 (Zen 3, AVX2), median of per-round ratios over 5 interleaved rounds against the build without this commit: classic lossless 4096x2048 full decode -9.9% at 1 thread, -10.5% / -13.5% / -12.4% at 4 / 8 / 16; classic 2 bpp -9.2% at 1 thread, -7.0% / -6.1% / -8.6%; classic 1024x1024 regions -7.0% to -8.5% at 1 thread; 18000x12000 classic overviews (reduce 2/3/4) -9.0% to -11.1%.
…in the x86-64 fused ICT pack After the centered rounding the fused pack's samples are whole numbers, yet the RGB8 store re-ran the generic quantization (floor, fraction compare, select) and the unsigned float-to-int conversion, which AVX2 emulates, and copied each 12-byte half through the stack. The fused pack now only clamps like the saturating u8 cast (NaN to zero) and converts with the native signed conversion (exact on 0..=255), and stores each half with one overlapping 16-byte store when the row has room past the eight pixels; the next pixels or the scalar remainder overwrite the spare bytes. Bytes past the planes' pixels are never written. Output is byte-identical. Measured on x86-64 (Zen 3, AVX2), median of per-round ratios over 5 interleaved rounds against the build without this commit: HT 2 bpp 4096x2048 full decode -7.7% at 1 thread and -5.3% at 4, HT 512x512 tiles -4.8%, classic 2 bpp -1.3% to -1.7%.
…atch Baseline x86-64 has no per-lane variable shift and no unsigned int-to-float conversion, so the classic midpoint dequantization loop stayed scalar on x86-64 while ARM64 vectorizes it. The row now runs inside the runtime dispatch on x86-64, where AVX2 vectorizes it. ARM64 is unchanged. Output is byte-identical. Measured on x86-64 (Zen 3, AVX2), median of per-round ratios over 5 interleaved rounds against the build without this commit: classic 2 bpp 4096x2048 full decode -4.4% at 1 thread, -4.3% at 4 and -7.2% at 16; classic 1024x1024 regions -1.5% to -1.8% at 1 thread.
…e the x86-64 SIMD dispatch Code blocks coded with the bypass, restart, causal or termination styles take the per-coefficient passes, which still compiled for baseline x86-64 (BSR-based `leading_zeros` on every MQ decision). They now run inside the runtime dispatch on x86-64 like the packed passes. ARM64 is unchanged. Output is byte-identical. Measured on x86-64 (Zen 3, AVX2) on a classic 2 bpp 4096x2048 RGB image encoded with Cmodes=BYPASS, median of per-round ratios over 5 interleaved rounds against the build without this commit: -7.3% at 1 thread, -6.0% at 4 and -3.5% at 16. Default code blocks (packed passes) are unchanged.
A decode_into called from outside the Rayon pool entered the pool once per parallel stage: each component's Tier-1, the IDWT and the MCT and pack were injected from the caller's thread, which slept until a pool thread had finished each of them. An Auto decode now enters the pool once, through rayon::scope, so its later stages start from a pool thread. It stays where it is when called from a pool thread, with a one-thread pool, with explicit parallelism (checked first, so it never starts the pool), with stage profiling on (a pool thread would keep its rows), and when its estimated serial time is under 1.25 times the fork cost: a decode only just worth forking spends much of its time in serial parts. A test-only switch keeps per-thread test counters on the test thread. Measured against the parent (built twice, as A and an A/A control) on an 18-core ARM64 machine, 5 interleaved rounds, median of per-round ratios, full-image RGB8 decodes: at 6 threads HT 512^2 -15%, HT lossless 256^2 -14% and 512^2 -13%, classic 192^2 -15% and 512^2 -9%, HT 4096x2048 mosaics -4% to -5%; at 18 threads HT lossless 512^2 -14%, HT 512^2 -11%, classic 512^2 -8% and 256^2 -5%, HT mosaics -5% to -8%, classic mosaics -2% to -3%. One-thread decodes, decodes that already run serially, region decodes and concurrent region decodes are unchanged within the A/A spread. Output and heap peak are unchanged.
The tile route queues one task per code-block row, largest first. Classic rows at coarse resolution levels are few and very unequal (an LL or level-4 row can carry many times the work of a level-1 row), so the queue ends on a long row while the other workers wait. Classic rows whose estimated work exceeds total / (8 x workers) are now split into runs of consecutive blocks of about equal work. The parts of a row share its output through a mutex: each block decodes into the worker's block scratch and is copied in under the lock (in a stale buffer each part first zeroes its own columns). HT rows stay whole. The extra queue entries, the mutex vector and one block of scratch per worker are charged to the decode budget up front. Unit tests cover the run splitting and the part counts. Interleaved A/B against the parent: full-image classic decodes are unchanged within +-1.5% at 6, 18 and default threads (their 79-94 rows per component already balance). On tile-route reduced-resolution decodes of an 18000 x 12000 classic image at 18 threads (measured with the band route switched off), 1/16 scale takes 15% less time and 1/8 scale 3.5% less. Output is byte-identical.
On the pool every IDWT level was its own pass of row strips, so the finer level waited at a barrier for the coarser level's slowest strip, and the levels' short phases left workers idle. The two finest levels `idwt::apply` reconstructs now run as one pass, `filter_2d_banded_pair`: each strip of the finer level first streams the rows of the coarser level it reads, with the halo rows of a strip of that level, into its worker's scratch, then streams its own rows from them. Its workers do not wait for the rest of the coarser level, and the coarser level is no longer written to the ping-pong buffer. - Those coarser rows go through the same expressions as in a pass over the whole level, so the output is bit-identical. The cost is the coarser level's halo rows, recomputed per strip of the finer level. - `RowSource` can skip band rows at its front, so a level can read its LL band rows from scratch (`Level2d::with_ll_rows`). - One thread, regions, the decoder hook and the integer path keep the level-by-level passes. HT 2 bpp full-image decode (4096x2048 RGB, an 18-core ARM64 machine): -1.5% at 6 threads, -4.7% at 18, -6.9% at the default; HT lossless -1.3%, -4.0%, -3.2%; one thread, regions and concurrent region decodes unchanged. Output is unchanged.
Batch workers enabled parallel referenced execution for HTJ2K plans only; classic plans never set the flag and the classic executor ignored it, so a prepared classic batch decoded every code block and wavelet level on one thread (4.7-8.3x slower than single-image decode). The HT row dispatch moves to a shared generic module (same types and sizes), and classic sub-bands now decode their rows of code blocks on the Rayon pool when every payload is one contiguous fragment, falling back to the serial path otherwise; IDWT steps follow the same flag. Extra classic workspaces are accounted and reserved like HT's, but bounded by the most code-block rows any sub-band holds, so small tiles keep few of them. On an 18-core ARM64 machine, decode only, byte-identical: batches of 2048 x 2048 RGB 191 -> 41 ms, 3584 x 3584 RGB 550 -> 104 ms, 4096 x 4096 gray 425 -> 51 ms, 4096 x 4096 RGB at half scale 191 -> 40 ms; 8 images in flight 1.7-1.9x faster; 64 small tiles unchanged. Peak heap +0.3% to +1.7%.
…ed plan BatchAllocationBudget::claim waited forever when a claim fit the execution cap but not the cap minus the budget's permanent baseline (the shared direct plan), because no claim can ever release those bytes. Such a claim now fails at once with AllocationTooLarge (cap = cap - baseline); claims that can fit still wait for other owners as before.
…s by plan Prepared batches had no byte admission, and an offset plan whose execution exceeds the per-decode bound (4096 x 4096 RGB, 6656 x 6656 gray, the 5661 x 8287 library scan) only failed once decode had started; an image whose own output exceeds the per-allocation cap aborted the whole decode call. Preparation now plans each offset plan's execution peak with the accounting its execution enforces (new hidden j2k-native planners, no allocation) and records a CPU-only choice on the prepared image; device decoders see the same codec plan as before. CpuBatchDecoder::prepare rejects, as indexed prepare errors, images whose serial execution exceeds the bound and images whose own output exceeds the cap; batches prepared for any decoder report the same error at decode before any decode work. decode_prepared admits every execution under the batch execution cap with one claim per worker at the largest plan it ran. Bytes and decode times unchanged where decodes succeeded before; oversized items fail in prepare instead of decode.
The staged CPU route keeps one whole-image scratch per window image until the window is materialized, eight images at a time, without any byte admission: eight 3584^2 RGB images held 4.3 GB of scratch. Each window now claims the sum of its images' planned execution peaks and ends early when the next image would push that sum past the batch execution cap, so in-flight staged scratch stays within the cap on hosts where it binds (four worst-case decodes on small hosts). Windows are not narrowed further: per-worker windows cut staged heap 75-80% but cost 10-33% decode time even with one worker. Unchanged on a 64 GB machine (cap 32 GiB).
…ayloads Parallel classic dispatch borrows every payload as one input slice and falls back to serial decode when a code block's bytes are split over fragments, as in multi-layer codestreams. Such plans still reserved the extra classic workspaces, which then sat unused: IIIF 512 tiles at 1/2 of a 5661x8287 8-layer scan grew peak heap 2.7%. Plans with a fragmented payload now plan and reserve no extra classic workspaces and no dispatch bytes; their wavelet steps stay parallel. Peak heap of that batch back to where it was before the parallel plans (399.7 MB), bytes unchanged.
On AArch64 the classic Tier-1 pass loops in run_arithmetic_coding_passes keep more values live than there are registers and reload spilled pairs with `ldp`: 16-byte loads at 8-byte-aligned stack offsets. The significance pass reloads one such pair (the zero-coding table and the context array pointers) once per symbol. Wherever the stack depth put one of these pairs across a 16 KiB page boundary, every reload was a page-split load and the passes ran 44-45% slower. That happened at three 16-byte positions per 16 KiB of stack depth: on the caller's thread (one-thread decodes) the caller's stack and environment chose them, on pool threads the build did. A 4 KiB-aligned local now makes the function realign its stack pointer to 4 KiB, so where the spills sit within a page no longer depends on the caller; here they are single loads and stores just below the anchor, and none can straddle a page. The frame grows by up to 12 KiB of stack per call, so builds without `std` keep the old frame. x86-64 is unchanged: its spills are single aligned registers, and the same stack-offset sweep finds no slow offset there. Output is byte-identical. Measured on an 18-core ARM64 machine at one thread, sweeping the caller's stack offset in 16-byte steps: the parent decoded the classic 2 bpp 4096x2048 mosaic in 246.6 ms at its worst offset against a 171.5 ms median (+44%), and a 512x512 classic tile +40% at its worst offset of a full 16 KiB period; with this change the medians are unchanged (171.5 and 4.57 ms) and no offset repeats as slow. Interleaved A/B over 7 rounds against two builds of the parent at 1, 6, 18 and default threads: classic mosaics -0.3% to +1.8% (A/A -1.6% to +1.2%), the 5661x8287 9/7 workload's full-ROI decode -1.2% to +1.8% (A/A -1.2% to +1.8%), heap unchanged, max RSS +0.3 MB at 18 threads (worker stack pages); no classic row of the decode benchmark set slower after 7- and 15-round reruns.
ImageTooLarge and AllocationTooLarge reached callers as the generic
J2kError::NativeDecode, so a decode that only exceeded the working-memory
cap looked like any other backend failure. j2k_native::DecodeError::classify
now reports DecodeErrorClass::ResourceLimit { requested, cap } for them,
and the facade maps that class to the new J2kError::ResourceLimit
{ requested, cap, source } on every decode route: J2kDecoder, the tile
batch functions and the owned batch API, as it already maps truncated
input and unsupported features. requested and cap are known for
AllocationTooLarge and None for ImageTooLarge. Host allocator failures
keep their classification. The source chain keeps the native error.
The native decode error conversion is now #[cold] and out of line. Inlined
into its 53 call sites, the larger match made HTJ2K lossless 4096x2048
decodes at 18 threads 1.0-1.4% slower in interleaved A/B runs (code
layout; a variant that kept J2kError at 96 bytes was 2.7% slower). Out of
line it measures 1.002 against base (A/A 1.000), and 0.995 / 0.997 on the
HTJ2K 2bpp and classic lossless rows.
Public additions (both enums are #[non_exhaustive]):
- j2k_native::DecodeErrorClass::ResourceLimit { requested, cap }
- j2k::J2kError::ResourceLimit { requested, cap, source }
J2kError grows from 96 to 112 bytes. Decoded bytes are unchanged.
On the pool, `idwt::apply` reconstructs the two finest levels of a component in one pass (`filter_levels_pair`), and each worker first streams the rows of the coarser level its strip reads into its band scratch. That scratch is the wider level's worker scratch plus those LL rows per concurrent strip, but the build plan and the planes floor charged one level's worker scratch only, so the LL rows went uncharged (at 4096 x 4096 with 4 workers, 8.6 MB). `apply` also still reserved an intermediate buffer for the second-finest level, which the paired pass reads from its sub-bands or the third-finest level and never writes. - One layout, `PairScratch`, sizes the paired band scratch for both the pass and `planned_pair_band_scratch_elements`, and one planning function, `idwt::planned_workspace_elements`, gives the plan and the planes floor the component's IDWT workspace: level by level, the finest level's output, the second-finest level's buffer and the finest level's band scratch; when the levels pair, the output, the third-finest level's buffer and the pair's band scratch. A serial decode or a code-block decoder hook keeps the level passes, so a component whose levels pair is planned for the larger of the two. - `apply` reserves the intermediate buffer for the level written before the last, which is the third-finest level below a paired pass. Where the LL rows are the larger term (small strips on many workers), the charge grows to what the IDWT holds; elsewhere it is unchanged and the IDWT holds less within it: 1024 x 768 gray on 4 threads holds 3.9 MB instead of 4.5 MB against a 4.1 MB charge, on 8 threads exactly its 4.2 MB charge. The pass itself is unchanged and output is bit-identical. On an 18-core ARM64 machine, 7 interleaved rounds at 1, 6, 18 and the default thread count: every full-image, scaled and 4096 x 2048 mosaic decode measured is within noise (median ratios 0.971 to 1.013, the same spread as between the 18-thread and default runs), with identical output. The mosaics, which pair their levels, peak at 1.5 MiB (1.3%) less heap. Tests: the planned paired band scratch equals what the pass grows to on pools of 2 to 18 threads; real paired decodes hold no more than the plan charges, keep only the level below the pair in the intermediate buffer and match the serial decode; and a paired tile's build is admitted exactly at the budget, refused before reserving anything one byte over, with the admitted decode's IDWT within its charge.
An x86-64 build targets baseline x86-64, which has no SSE4.1 rounding or FMA instructions, so outside a runtime SIMD dispatch every `round_ties_even` is a call to `rintf` and every `mul_add` a call to `fmaf`. The centered rounding after the inverse ICT was such a scalar loop: one library call per sample. - `round_centered_and_shift` runs at the CPU's SIMD level on x86-64 (eight samples per AVX2 instruction). - The last-level RGB8 pack fuses the inverse ICT, the rounding and the interleave at the SIMD level on x86-64, as ARM64's scalar loop already does through auto-vectorization. The fused pack is kept out of line so the per-row emit closure still inlines into the dispatched row streamer (an outlined closure loses the dispatch's target features). Output is byte-identical. Measured on x86-64 (Zen 3, AVX2), median of per-round ratios over 5 interleaved rounds against the base build: HT 2 bpp 4096x2048 full decode -31.1% at 1 thread and -15.5% at 16, HT 512x512 tiles -25.4%, 18000x12000 HT overview (reduce 2) -18.7% at 1 thread, HT 1024x1024 region -12.1%, classic 2 bpp -12.2% at 1 thread. Lossless rows unchanged (within the A/A spread).
…packed passes Classic code-blocks coded with selective arithmetic coding bypass or per-pass termination, which split into several codeword segments, now take the packed-column Tier-1 passes, as single-segment blocks already did (context resets and segmentation symbols included, in either case). Raw (bypass) segments get packed cleanup, significance propagation and magnitude refinement passes that read the same bits as the per-coefficient raw passes, and the arithmetic segments of such blocks run the existing packed arithmetic passes. Only vertically causal blocks keep the per-coefficient passes. A test-only switch forces the per-coefficient passes. The parity test decodes blocks of all sixteen non-causal style combinations, sizes from 1x1 to 64x64 and corrupted copies of them both ways, and requires the same samples or the same error, in strict and lenient modes. Based on the prototype in #311. Co-authored-by: Justin Coyne <jcoyne@justincoyne.com>
In the classic cleanup pass, a column of four insignificant, unvisited coefficients decodes one run-length decision. While that context's MPS is zero and the MQ decoder needs no renormalization, each such decision returns zero and changes nothing but A, which drops by Qe. From the current registers, ArithmeticDecoder::mps_run gives how many decisions stay on that path (A - k * Qe at least 0x8000 and above the upper half of C), and the pass skips that many consecutive all-zero columns with one subtraction. Decisions, registers and context states are exactly those of decoding each column; a unit test compares the registers after every skip with per-decision decoding over adapting and fixed contexts. Classic full-image decode, interleaved A/B against the parent: 2 bpp -6.7% at 1 thread, -5.8% at 6, -6.4% at 18; lossless -2.3% at 1 thread and -3.5% at 18. Output is byte-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This series speeds up CPU decoding in the Tier-1, packet-parsing, IDWT and prepared-batch paths, and reports memory-limit failures through a dedicated error class. It follows #313; successful decoded pixels remain byte-identical.
Decode changes
Representative interleaved measurements on ARM64: BYPASS full decoding takes 0.78x the previous time at one thread; HT lossless full decoding improves about 10.5% at one thread; a batch of 2048×2048 RGB images takes 41 ms instead of 191 ms on an 18-core machine, with 0.3–1.7% more peak heap. Kernel parity and output comparisons cover the affected paths.
API and error behavior
J2kError::ResourceLimit { requested, cap, source }andDecodeErrorClass::ResourceLimit { requested, cap }to existing non-exhaustive enums. Existing variant discriminants are preserved.ImageTooLargeandAllocationTooLargefailures use this class on decode routes and JP2/JPH metadata inspection. The display prefix changes todecode resource limit exceeded:; the native source remains available. Host allocator failures keep their previous classification.CpuBatchDecoderpreparation records oversized individual outputs or planned executions as indexed errors and excludes those items from prepared groups. Backend-neutral preparation retains its existing behavior. On either path, combined group output exceeding the allocation cap still fails the whole decode call.referenced_classic_plan_execution_bytesandreferenced_htj2k_plan_execution_bytes.Shared-file coordination
As discussed in #133, the packet/allocation/setup changes touch contributor-owned code in
j2c/build.rs,j2c/build/allocation/plan.rs,j2c/build/selection.rs,j2c/segment.rs,j2c/segment/classic.rs,j2c/tag_tree.rs,j2c/tile.rs,j2c/tile/cursor.rs,j2c/decode/state.rs,j2c/decode/subband/{classic,tile_parallel}.rs,j2c/idwt.rs,j2c/idwt/orchestrate.rs,j2c/decode/store.rs,color/mod.rs,image.rs,lib.rsandtests.rs, all undercrates/j2k-native/src/.The shared band provider and streamed line-reader implementation are unchanged. Their later work remains separate for coordination.
Validation
target=all,mode=quick).