perf: optimize quantization and sampled rendering

[taskylizard]

No description provided.

[codspeed-hq]

Merging this PR will improve performance by 37.25%

⚡ 5 improved benchmarks
✅ 17 untouched benchmarks
⏩ 22 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	WallTime	build_palette	1,172.2 µs	656.3 µs	+78.6%
⚡	WallTime	map_frame_warm_cache	391.1 µs	314.8 µs	+24.25%
⚡	WallTime	composite_sampled	4.4 ms	3.8 ms	+15.68%
⚡	WallTime	gif_animation	197.5 ms	179.5 ms	+10.08%
👁	WallTime	map_frame_cold_cache	1,322.6 µs	767.3 µs	+72.38%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing perf/maple-core-hot-paths (c46a929) with trunk (e514776)}

Footnotes

1. 22 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[taskylizard]

@codspeedbot fix this regression

[codspeed-hq]

@codspeedbot fix this regression

Fixed in 998b345.

Root cause. The WallTime flame graphs point squarely at Quantizer::update_box: it costs 642 µs on trunk and 1.10 ms on this branch, and 632 µs of that increase is i32::min.

The single-pass scan added in "scan color boxes once" widened all six box bounds inside the per-cell loop, so every occupied histogram cell paid six dependent min/max updates. That serial chain ended up costing more than the extra passes it removed.

Fix. The scan still runs once, but the bookkeeping moved out of the inner loop:

• the inner loop is now a plain count of occupied cells, which the compiler can vectorize;
• c0/c1 bounds are widened once per non-empty row instead of once per cell;
• c2 bounds only need the first and last occupied cell of a row, and are skipped entirely once the bound already reaches the edge of the box.

Correctness. Palette output is unchanged — quantizing a fixed image yields byte-identical palette and indexed output on trunk, on the previous branch head, and after this change. The existing test suite passes.

Measured locally on the codspeed profile (build_palette, fastest / median):

Variant	Fastest	Median
trunk	407.7 µs	415.8 µs
branch before this commit	314.3 µs	328.7 µs
branch after this commit	275.2 µs	285.7 µs

CPU Simulation instrumentation agrees: 2.30 ms on trunk → 1.89 ms before → 1.79 ms after.

The next CodSpeed run on the macro runners will confirm the regression is cleared.

Summary

• Warming up the performance wizard...
• Investigating build_palette regression — Root cause found: update_box widens six box bounds inside the per-cell scan loop
• Fixing the update_box hot loop — Bounds bookkeeping hoisted out of the per-cell scan; build_palette now faster than trunk locally and in simulation
• Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}

[taskylizard]

@codspeedbot any other performance improvements you can see? just asking, don't implement anything yet

[codspeed-hq]

@codspeedbot any other performance improvements you can see? just asking, don't implement anything yet

Yes — there's still a fair amount on the table. I profiled the current head (998b345) and read the hot paths; no code changed, this is just the shortlist.

Where the time actually goes now

Benchmark	Dominant cost	Owner
build_palette (999 µs)	prescan_quantize 56% self, update_box subtree 34%	user
map_frame_dithered (3.4 ms)	quantize_fs_dither 91% self	user
map_frame_cold_cache (1.3 ms)	find_best_colors 56%, find_nearby_colors 9%	user
composite_multi_layer (8.5 ms)	sample_linear_opaque 48% self (~65% with its inlined indexing), Render::add closure 25%	user
webp_* (40–570 ms)	libwebp FTransform/ITransform/GetResidualCost/QuantizeBlock	third party
load_photo (35 ms)	zune-jpeg progressive decode (95%)	third party
gif_serialize (47 ms)	weezl LZW (Tree::at_key/iterate, ~19% of gif_animation_dithered)	third party

Notably, the last PR fixed update_box, and that moved the bottleneck: prescan_quantize is now the single biggest item in build_palette.

Ranked ideas

1. Occupancy bitmap for the median cut — biggest structural win in build_palette
HIST_C2_ELEMS is exactly 32, so the occupancy of one (c0, c1) row is exactly one u32 — the whole 128 KB histogram condenses to a 8 KB bitmap that stays in L1. Build it once with a linear pass after the prescan (vectorizable, ~tens of µs), then update_box becomes: count = (word & mask).count_ones(), c2min = trailing_zeros, c2max = 31 - leading_zeros. That replaces the current per-row filter().count() + position() + rposition() scans (~34% of the benchmark) with three register ops per row. The histogram is immutable during select_colors, so the bitmap stays valid across all 255 splits. compute_color can use it to skip empty rows too.

2. Batch the histogram scatter in prescan_quantize
For a 400×300 frame this costs ~6 ns/pixel, which is far more than the arithmetic — it's the dependent read-modify-write into a 128 KB (L2-resident) table. Two cheap changes: compute indices for a small batch (8 pixels) first, then do the increments, so the misses overlap instead of serializing; and replace if *cell < u16::MAX { *cell += 1 } with a branchless saturating_add(1).

3. Shrink the inverse-colormap footprint
The histogram doubles as the inverse colormap, storing palette_index + 1 in a u16 — 128 KB of random access in quantize_no_dither/quantize_fs_dither. A u8 colormap (64 KB) plus a validity bitmap (8 KB, L1-resident) halves the traffic in the two hottest mapping loops.

4. find_best_colors: i32 instead of i64, and reuse the scratch buffers
Max weighted squared distance here is ~910 k, so i32 is plenty — that halves the traffic on bestdist and makes the 128-cell min-reduce SIMD-friendly. Storing bestcolor as i32 in the loop and narrowing to u8 at the end would let LLVM vectorize the compare-and-select cleanly. Also, bestdist/bestcolor/colorlist/mindist are heap-allocated on every fill_inverse_cmap call; hoisting them into Quantizer fields (or stack arrays) removes that per-call allocate-and-zero.

5. quantize_fs_dither: hoist the row slices
~28 ns/pixel is a lot even for a serial error-diffusion loop. It calls img.get_pixel(x, y) per pixel (pixel_indices + bounds check show up at ~6% on their own) and does output[row * width + x] plus ~7 bounds-checked self.fserrors[..] accesses per pixel. Taking a &[u8] row slice of the source, a &mut [u8] row slice of the output, and a local &mut [i16] for the error buffer before the column loop should strip most of that without touching the algorithm.

6. sample_linear_opaque: fewer bounds checks, and skip the ×255 / ÷255 round-trip
It's the top user-code function in the whole suite. The interior fast path does 12 individually bounds-checked raw[..] loads even though the guard already proves they're in range — loading two 8-byte sub-slices (first_chunk::<8>() on the two rows) collapses that to one check per row and vectorizes the four channels. Separately, in the opaque interior case every channel is multiplied by a constant 255.0 and then divided out again by postblend(sc) in Render::add; when all 9 taps are interior the two cancel. Worth ~36 multiplies + 3 divides per pixel — but it changes results by a few ULP, so it needs the existing opaque_sampler_matches_general_premultiplied_sampler test to be relaxed to a tolerance, which is a call for you rather than me.

7. Render::add: monomorphize on input_opaque
The if input_opaque branch sits inside the per-pixel loop even though it's loop-invariant. Splitting the row closure over a const generic (or a tiny generic helper taking the sampler as a type parameter) removes the branch and lets each sampler inline into a tighter loop.

Not worth chasing

webp_serialize, webp_animation, webp_single_frame* and load_photo are ~95%+ inside libwebp and zune-jpeg. The only lever there is configuration, not Rust code — e.g. config.thread_level is never set in webp_anim.rs, and enabling it typically buys 10–20% on lossy method = 4. That said, the batch path already parallelizes frames with rayon, so it would only help the single-frame encoders and risks oversubscription elsewhere. gif_serialize is weezl's LZW encoder; nothing to do short of swapping the encoder.

One caveat on all of the above: benches/common/mod.rs pins RAYON_NUM_THREADS=1, so anything that wins purely by adding parallelism (e.g. a parallel histogram prescan) would be invisible in these numbers even if it helps in production.

Happy to implement any subset — just say which ones and I'll do them one commit at a time so each shows up separately in the report.

Summary

• Warming up the performance wizard...
• Warming up the performance wizard... — Reading the request and preparing the analysis
• Scouting for optimization opportunities — Profiled the 22 benchmarks and read the hot paths in quantize.rs, pixer.rs and render.rs
• Handling comment — Reported optimization opportunities; no code changes made, as requested
• Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}

[taskylizard]

@codspeedbot Implement 1, 2, 3, 4, 5, 7. Do them one commit at a time

[codspeed-hq]

@codspeedbot Implement 1, 2, 3, 4, 5, 7. Do them one commit at a time

Done — one commit per idea, four of the six kept. Ideas 3 and 7 were implemented, measured, and reverted: both are regressions on this code. Details below.

Kept

Measured on the codspeed profile with RAYON_NUM_THREADS=1, median of 100 samples, each line against the commit before it:

#	Commit	Benchmark	Before	After
1	perf(quantize): drive the median cut from an occupancy bitmap	build_palette	284.7 µs	217.2 µs
2	perf(quantize): split the histogram scatter into two lanes	build_palette	217.2 µs	202.1 µs
4	perf(quantize): build the inverse colormap in i32 with stack scratch	map_frame_cold_cache	489.6 µs	421.2 µs
5	perf(quantize): walk the mapping loops over row slices	map_frame_warm_cache	148.2 µs	141.1 µs
		map_frame_dithered	1.346 ms	1.323 ms

Cumulative, against the previous branch head:

Benchmark	998b345	now
build_palette	284.7 µs	202.4 µs (−29%)
map_frame_cold_cache	521.2 µs	384.9 µs (−26%)
map_frame_warm_cache	148.5 µs	141.1 µs (−5%)
map_frame_dithered	1.360 ms	1.316 ms (−3%)

1 — occupancy bitmap. Exactly as sketched: the 128 KB histogram condenses to an 8 KB bitmap. update_box turned out to work best fully branch-free — each row is one masked load, a popcount and two ORs, and the six box bounds are read off c0/c1/c2 occupancy words after the scan (trailing_zeros / leading_zeros) rather than widened as it goes. compute_color walks the set bits, so it never touches an empty cell.

2 — histogram scatter. The batching part of this one did not work: staging 8 (or 4) indices before the increments made build_palette slower (216 µs → 234 µs), because the increments still serialize — the compiler cannot prove the batched indices are distinct, so it only pays for staging them. The real problem is that neighbouring pixels of a photo land in the same cell, so the scan is one long store-to-load-forwarded chain. Scattering even and odd pixels into two tables and folding them back in a linear pass fixes that: 216 µs → 202 µs. Four lanes was worse than two. Branchless saturating_add also measured slower than the existing if *cell < u16::MAX (225 µs vs 215 µs), so it stayed.

4 — i32 + scratch reuse. Both halves paid off, and this is the largest single win after the bitmap.

5 — dither row slices. Applied to quantize_no_dither as well, which is where most of it landed (map_frame_warm_cache −14% on the fastest sample). Hoisting the error-limiter table and the palette out of self the same way made no difference, so they were left alone.

Measured, not kept

3 — u8 colormap + validity bitmap. This is a regression, and the reasoning in my earlier comment was wrong. Today the u16 sentinel (palette_index + 1, 0 = not cached) means the "is this cached?" test is fused into the value you already loaded. Splitting into a 64 KB u8 table plus an out-of-band validity bitmap makes it two lookups per pixel instead of one — and the box-granularity bitmap is only 64 bytes, so the traffic saving is real but irrelevant: map_frame_warm_cache 148 µs → 194 µs (+30%), map_frame_cold_cache 481 µs → 517 µs. I also tried a variant with the colormap re-indexed so a box occupies one contiguous 128-byte block (validity check becomes index >> 7); still not competitive. The 128 KB table is comfortably L2-resident here, so halving it buys nothing.

7 — monomorphize Render::add on input_opaque. Also a regression, in both forms I tried:

Variant	composite_sampled	composite_multi_layer
current (if input_opaque in the loop)	2.441 ms	4.163 ms
generic over the sampler F	2.565 ms	4.380 ms
const OPAQUE: bool	2.628 ms	4.484 ms

The branch is loop-invariant, so it is perfectly predicted and effectively free; what the split actually does is duplicate a very large loop body, and the instruction-cache cost outweighs it. Worth noting for 6 too, if you ever want it: that one is about arithmetic, not branching, so it does not share this problem — but it still needs the opaque_sampler_matches_general_premultiplied_sampler test relaxed to a tolerance, which is your call.

Correctness

Palette bytes and indexed output are byte-identical to 998b345 across three template/photo combinations (toaster frames 11/0/3 with frog.jpg/monkey.jpg/dolphin.jpg), for the palette, the no-dither, the warm-cache repeat and the dithered paths. cargo fmt, cargo clippy -D warnings, both test suites (including --no-default-features), the wasm target check and a build against the declared MSRV (1.88) all pass, and the benchmarks were run end to end through the CodSpeed runner.

The next CodSpeed run on the macro runners will confirm the numbers.

Summary

• Warming up the performance wizard...
• Warming up the performance wizard... — Comment handling finished successfully
• Building the baseline — Workspace builds, 19 tests pass, and the quantize/render benchmarks are measured as a reference
• 1/6 — Occupancy bitmap for the median cut — Committed: build_palette 284.7 µs → 217.2 µs median, palette output byte-identical
• 2/6 — Batch the histogram scatter — Committed a two-lane scatter instead: build_palette 217.2 µs → 202.1 µs median (plain batching measured as a regression)
• 3/6 — Shrink the inverse colormap — Measured and rejected: the u8 colormap + validity bitmap costs a second dependent lookup per pixel and makes map_frame_warm_cache 30% slower. Reverted, no commit.
• 4/6 — find_best_colors in i32 with reused scratch — Committed: map_frame_cold_cache 489.6 µs → 421.2 µs median, output byte-identical
• 5/6 — Hoist the row slices in the dither loop — Committed: map_frame_warm_cache 148.2 µs → 141.1 µs, map_frame_dithered 1.346 ms → 1.323 ms
• 6/6 — Monomorphize the compositor on the sampler — Measured and rejected: both a generic-sampler and a const-generic split make composite_sampled 5-8% slower. Reverted, no commit.
• Validating the result — Tests, clippy, fmt, MSRV 1.88 and a CodSpeed runner pass all clean; 4 commits pushed
• Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}

[taskylizard]

@codspeedbot There's a regression, can you look into it?

[codspeed-hq]

@codspeedbot There's a regression, can you look into it?

Found it, and it is my mistake from the previous round. Fixed in c46a929, and the CodSpeed run on the macro runners has already confirmed it:

Benchmark	trunk	previous head	now
map_frame_cold_cache	1,322.6 µs	1,632.8 µs	767.3 µs (−42% vs trunk, 2.1× vs previous head)

No regressions left on the PR; build_palette (+78.6%), map_frame_warm_cache (+24.3%), composite_sampled (+15.7%) and now gif_animation (+10.1%) are all improved against trunk.

Root cause

The regression came from "build the inverse colormap in i32 with stack scratch", and specifically from the part I claimed was a win: putting the winning color in a second i32 array next to the i32 distances inside find_best_colors.

That loop compares each of the 128 cells of a box against the running best distance, and after the first candidate color the compare almost never wins — a nearly perfectly predicted, almost never taken branch. Once both arrays were i32, LLVM vectorized the update: the branch became an unconditional load, compare, select and store on every cell, for both arrays.

The reason I did not catch it is that this repository is benchmarked in walltime mode on codspeed-macro, which is ARM64, while I measured locally on x86_64. The same source compiles to two qualitatively different loops (-C target-cpu=generic, per-cell instruction mix):

Variant	x86_64	aarch64
previous head (i32 + parallel index array)	SSE shuffle/extract heavy	dup/cmgt/bsl — vectorized select
i64 + direct u8 store	scalar cmpq + jge	scalar cmp + b.ge

On the Ice Lake box I measured on, the shuffle-and-extract version is free; on the Neoverse macro runners it cost 460 µs in find_best_colors alone (842 µs → 1.3 ms in the flame graphs), which was the entire regression.

Fix

find_best_colors goes back to i64 distances with the winner stored straight into the u8 output, which keeps the loop scalar and branchy on both targets. The comment in the code now records why, so the i32 rewrite does not come back. Everything else from that commit is kept, because it is a measured win on the macro runners:

• find_nearby_colors in i32: 140 µs → 70 µs there;
• the stack scratch, so bestdist no longer allocates per call — which is why find_best_colors is now 341 µs, well below the 842 µs it cost before the bad commit.

Correctness

Palette, no-dither, warm-cache-repeat and dithered bytes are identical to the previous head for toaster frames 11/0/3 over frog.jpg/monkey.jpg/dolphin.jpg. cargo fmt, clippy -D warnings and the test suites pass, and the CI lint/test job is green.

For future rounds I will validate hot-loop changes against the aarch64 codegen and a CI run rather than trusting local x86 walltime — I have noted that for this repository.

Summary

• Warming up the performance wizard...
• Warming up the performance wizard... — Confirmed the regression comes from the last four commits
• Investigating the map_frame_cold_cache regression — Root cause: the i32 rewrite made LLVM vectorize the per-cell compare-and-select, which is a loss on the ARM64 macro runners
• Restoring the scalar compare-and-store loop — Committed; output byte-identical and no cost on x86
• Confirming on the macro runners — Regression cleared: map_frame_cold_cache 1,632.8 µs -> 767.3 µs, now 42% faster than trunk
• Processing benchmark results — Benchmarks have been successfully run and processed. The results are available in Codspeed
• Handling comment — Comment handling finished successfully
• Handling comment — Comment handling finished successfully

_{Learnings updated. See what CodSpeed remembers about this repository.}

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}