diff --git a/docs/known-gaps.md b/docs/known-gaps.md
index 9e2013b..20201e1 100644
--- a/docs/known-gaps.md
+++ b/docs/known-gaps.md
@@ -53,7 +53,7 @@ HEIF 解码也已调优（M3），基准项目是 `Sdcb.SimdCodecs.Heif.Benchmar
0..255（线性光数据如增益图 HDR 再加 `srgbEncode: true`），但不做色调映射。
- **多帧读取器的帧数可能少于文件里的帧数。** `ImageInfo.FrameCount` 报告文件里有多少帧或页，但：
- APNG：解码还没实现，读取器只给出 PNG 的默认图（1 帧）；
- - JBIG2：独立 `.jb2` 文件只解码第 1 页，读取器只有 1 帧。
+ - JBIG2：门面的读取器只给出独立 `.jb2` 文件的第 1 页（1 帧）；其他页用 `Jbig2Decoder.DecodeFile(file, pageNumber)`。
- **接口里没有“时间戳 → 帧号”的辅助方法。** WebP 包的 `WebpAnimationDecoder.FrameIndexAt` 有，接口层没有；
需要时可以按各帧的 `Duration` 自己累加。
- **`IImageFrameReader` 和 `ImageDecoder` 会话都不是线程安全的**（动画需要在调用之间保留画布状态），每个线程各开一个。
diff --git a/docs/known-gaps/jbig2.md b/docs/known-gaps/jbig2.md
index 2086601..9c02af3 100644
--- a/docs/known-gaps/jbig2.md
+++ b/docs/known-gaps/jbig2.md
@@ -2,10 +2,19 @@
条目约定（“不支持”“门面不提供”“未调优”）和跨格式的门面限制见 [总览](../known-gaps.md)。
-- **独立 `.jb2` 文件只解码第 1 页。** PDF 内嵌的 JBIG2 流没有文件头可供识别，需要直接调用
- `Jbig2Decoder.Decode(embedded, globals, width, height)`，门面不处理。
+- **门面只解码独立 `.jb2` 文件的第 1 页。** 其他页用 `Jbig2Decoder.DecodeFile(file, pageNumber)`，页数由
+ `Jbig2Decoder.TryReadFileInfo(file, out w, out h, out pages)` 给出。PDF 内嵌的 JBIG2 流没有文件头可供识别，
+ 需要直接调用 `Jbig2Decoder.Decode(embedded, globals, width, height)`，门面不处理。
- **不支持**：
- EXTTEMPLATE 通用区域；
- - 使用 HENABLESKIP 的半色调区域；
- - MMR 编码的半色调区域和图案字典。
+ - 引用或保留算术上下文的符号字典（符号字典标志的 bit 8、9）；
+ - MMR 的 2D 扩展码（含 T.6 非压缩模式）。
+- **Huffman 编码只有部分覆盖。** 标准表、自定义表段、未压缩和 MMR 的高度类集合位图、Huffman 文本区域，都有
+ `Jbig2ExtendedOracleTests` 用手工组装的码流对照 jbig2dec。Huffman 符号字典里的精细化 / 聚合符号（SDHUFF 与
+ SDREFAGG 同时为 1）、Huffman 文本区域里的逐实例精细化，没有黑盒编码器能造流，只有 issue #1 语料里的
+ `042_14.jb2` 用到，而 jbig2dec 和本解码器都判它为坏流，所以这两条路径没有经过对照验证。标准表 B.7 的码长
+ 之和是 15/16（不满），也只在单元测试里检查过。
+- issue #1 附件里的 114 个 `.jb2` 文件（来源和许可不明，不进仓库）由 `Issue1Jbig2CorpusTests` 在本地回归：
+ 设置 `SIMDCODECS_ISSUE1_DIR` 指向解压目录即可。112 个和 jbig2dec 逐页逐像素一致；`042_13`、`042_14` 两个
+ jbig2dec 和本解码器都在同一处判为坏流，本解码器抛 `InvalidDataException`。
- 未调优。
diff --git a/docs/known-gaps/jpeg2000.md b/docs/known-gaps/jpeg2000.md
index f477229..13a9245 100644
--- a/docs/known-gaps/jpeg2000.md
+++ b/docs/known-gaps/jpeg2000.md
@@ -2,14 +2,22 @@
条目约定（“不支持”“门面不提供”“未调优”）和跨格式的门面限制见 [总览](../known-gaps.md)。
-- **只支持 Part 1（T.800），定位是 PDF 里的 `/JPXDecode`。**
+- **只支持 Part 1（T.800），定位是 PDF 里的 `/JPXDecode`。** 多 tile、tile-part、自定义 precinct、
+ 五种进度顺序、SOP / EPH 标记、JP2 调色板（`pclr` + `cmap`）都已支持。
- **不支持**：
- 进度顺序变更（POC）；
- 打包的包头（PPM、PPT）；
- 感兴趣区域（RGN / MAXSHIFT）；
- - 在 tile-part 头里覆盖主头参数的标记。
-- **分量位深最多 16 位，且所有分量位深必须相同。**
+ - 在 tile-part 头里覆盖主头参数的标记（COD / COC / QCD / QCC）；
+ - code-block 样式里的选择性旁路（0x01）、每个 pass 单独终止（0x04）、垂直因果上下文（0x08）。
+ 上下文重置（0x02）、可预测终止（0x10）、分段符号（0x20）已支持；
+ - JP2 的通道定义盒（`cdef`）；
+ - 分量子采样、带符号分量。
+- **分量位深最多 16 位，且所有分量位深必须相同**；经调色板映射后的各通道位深也必须相同。
- **门面只呈现 1 或 3 个分量、且不是 sYCC 的图像**；带 alpha、CMYK、sYCC 的图门面返回 `false`，
`Jpeg2000Decoder.Decode` 仍会返回全部分量。
-- **没有分辨率缩减（reduce）解码。**
-- **已有基准测试，性能待调优**：基准项目 `Sdcb.SimdCodecs.Jpeg2000.Benchmarks` 已建立并在 `results/` 记录初始基线；目前纯托管单核解码耗时约为 libopenjp2 纯 C 核心的 1.9×~2.2×；EBCOT 条带位掩码快速跳过、Code-block 级多线程并发与系数重构缓冲区复用尚未实现。
+- **门面不提供分辨率缩减（reduce）**；`Jpeg2000Decoder.Decode(data, reduce)` 可以直接调用。
+- **真实文件回归**：issue #1 附带的 16 个 JPEG 2000 文件（来源和许可不明，不进仓库）全部与
+ `opj_decompress` 一致（5/3 逐样本相同，9/7 最多差 1）。改动解码器后按 `Issue1Jpeg2000CorpusTests`
+ 的说明下载并跑一遍。
+- **已有基准测试，性能待调优**：基准项目 `Sdcb.SimdCodecs.Jpeg2000.Benchmarks` 已建立并在 `results/` 记录初始基线；目前纯托管单核解码耗时约为 libopenjp2 纯 C 核心的 1.9×~2.2×；EBCOT 条带位掩码快速跳过、Code-block 级多线程并发与系数重构缓冲区复用尚未实现。tile 之间是串行解码的。
diff --git a/src/Sdcb.SimdCodecs.Jbig2/GenericRegionDecoder.cs b/src/Sdcb.SimdCodecs.Jbig2/GenericRegionDecoder.cs
index 9250695..3d34bf9 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/GenericRegionDecoder.cs
+++ b/src/Sdcb.SimdCodecs.Jbig2/GenericRegionDecoder.cs
@@ -97,6 +97,11 @@ internal static class GenericRegionDecoder
///
/// The decode's remaining pixel allowance, charged before anything is allocated.
///
+ ///
+ /// USESKIP / SKIP of T.88 §6.2.5.7: where this bitmap holds a 1 the pixel is not
+ /// coded at all and is set to 0. Null when USESKIP is 0. Must be
+ /// x .
+ ///
public static Jbig2Bitmap Decode(
ref MqDecoder mq,
scoped Span contexts,
@@ -105,8 +110,11 @@ public static Jbig2Bitmap Decode(
int template,
bool typicalPrediction,
scoped ReadOnlySpan at,
- Jbig2PixelBudget budget)
+ Jbig2PixelBudget budget,
+ Jbig2Bitmap? skip = null)
{
+ if (skip is not null && (skip.Width != width || skip.Height != height))
+ throw new ArgumentException("The SKIP bitmap must match the region size.", nameof(skip));
var expectedAt = template == 0 ? 8 : 2;
if (at.Length < expectedAt)
throw new ArgumentException($"GBTEMPLATE {template} needs {expectedAt / 2} AT pixel(s).", nameof(at));
@@ -138,8 +146,22 @@ public static Jbig2Bitmap Decode(
}
var row = y * width;
- for (var x = 0; x < width; x++)
- bitmap.Data[row + x] = (byte)mq.Decode(contexts, Context(bitmap, x, y, template, at));
+ if (skip is null)
+ {
+ for (var x = 0; x < width; x++)
+ bitmap.Data[row + x] = (byte)mq.Decode(contexts, Context(bitmap, x, y, template, at));
+ }
+ else
+ {
+ // §6.2.5.7 step 3 c): a skipped pixel costs no decision and reads as 0.
+ // The bitmap is freshly zeroed, so leaving it alone is setting it to 0.
+ var skipped = skip.Data;
+ for (var x = 0; x < width; x++)
+ {
+ if (skipped[row + x] != 0) continue;
+ bitmap.Data[row + x] = (byte)mq.Decode(contexts, Context(bitmap, x, y, template, at));
+ }
+ }
}
return bitmap;
diff --git a/src/Sdcb.SimdCodecs.Jbig2/HalftoneDecoder.cs b/src/Sdcb.SimdCodecs.Jbig2/HalftoneDecoder.cs
index 1cf007c..a63ffee 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/HalftoneDecoder.cs
+++ b/src/Sdcb.SimdCodecs.Jbig2/HalftoneDecoder.cs
@@ -56,6 +56,23 @@ public static Jbig2Bitmap[] DecodePatternDictionary(
ref mq, contexts, count * patternWidth, patternHeight, template,
typicalPrediction: false, CollectiveAt(patternWidth), budget);
+ return SlicePatterns(collective, count, patternWidth, patternHeight);
+ }
+
+ ///
+ ///
+ /// for HDMMR = 1: the collective bitmap is MMR-coded (T.88 §6.7.5, Table 27).
+ ///
+ public static Jbig2Bitmap[] DecodePatternDictionaryMmr(
+ ReadOnlySpan data, int patternWidth, int patternHeight, int maxIndex, Jbig2PixelBudget budget)
+ {
+ var count = maxIndex + 1;
+ var collective = MmrDecoder.Decode(data, count * patternWidth, patternHeight, budget);
+ return SlicePatterns(collective, count, patternWidth, patternHeight);
+ }
+
+ private static Jbig2Bitmap[] SlicePatterns(Jbig2Bitmap collective, int count, int patternWidth, int patternHeight)
+ {
var patterns = new Jbig2Bitmap[count];
for (var i = 0; i < count; i++)
patterns[i] = collective.Crop(i * patternWidth, 0, patternWidth, patternHeight);
@@ -75,6 +92,8 @@ public static Jbig2Bitmap[] DecodePatternDictionary(
/// HTEMPLATE, the generic template each bitplane uses.
/// HDEFPIXEL.
/// HCOMBOP.
+ /// HMMR: the grey-scale bitplanes are MMR-coded.
+ /// HENABLESKIP: grid cells whose pattern falls wholly outside the region are not coded.
internal readonly record struct HalftoneParameters(
int Width,
int Height,
@@ -86,22 +105,47 @@ internal readonly record struct HalftoneParameters(
int VectorY,
int Template,
byte DefaultPixel,
- CombinationOperator Combination);
+ CombinationOperator Combination,
+ bool Mmr = false,
+ bool EnableSkip = false);
/// Decodes a halftone region (T.88 §6.6.5).
- /// The arithmetic decoder, positioned at the grey-scale bitplanes.
+ /// The coded grey-scale bitplanes, after the segment's fixed fields.
/// The region parameters, read from the segment header.
/// The dither patterns from the pattern dictionary this region names.
/// The decode's remaining pixel allowance.
public static Jbig2Bitmap DecodeHalftoneRegion(
- ref MqDecoder mq, HalftoneParameters p, Jbig2Bitmap[] patterns, Jbig2PixelBudget budget)
+ ReadOnlySpan data, HalftoneParameters p, Jbig2Bitmap[] patterns, Jbig2PixelBudget budget)
{
if (patterns.Length == 0)
throw new InvalidDataException("JBIG2: halftone region refers to an empty pattern dictionary.");
budget.Charge(p.Width, p.Height);
var region = new Jbig2Bitmap(p.Width, p.Height, p.DefaultPixel);
- var grey = DecodeGrayscale(ref mq, p.GridWidth, p.GridHeight, patterns.Length, p.Template, budget);
+
+ // §6.6.5.1: with HENABLESKIP, a grid cell whose pattern would land wholly
+ // outside the region is not coded in any bitplane. The test is on the
+ // pattern's top-left corner, in whole pixels, against the pattern size.
+ Jbig2Bitmap? skip = null;
+ if (p.EnableSkip)
+ {
+ budget.Charge(p.GridWidth, p.GridHeight);
+ skip = new Jbig2Bitmap(p.GridWidth, p.GridHeight);
+ var patternWidth = patterns[0].Width;
+ var patternHeight = patterns[0].Height;
+ for (var m = 0; m < p.GridHeight; m++)
+ {
+ for (var n = 0; n < p.GridWidth; n++)
+ {
+ var x = (p.GridX + (long)m * p.VectorY + (long)n * p.VectorX) >> 8;
+ var y = (p.GridY + (long)m * p.VectorX - (long)n * p.VectorY) >> 8;
+ if (x + patternWidth <= 0 || x >= p.Width || y + patternHeight <= 0 || y >= p.Height)
+ skip.Data[m * p.GridWidth + n] = 1;
+ }
+ }
+ }
+
+ var grey = DecodeGrayscale(data, p, patterns.Length, skip, budget);
for (var m = 0; m < p.GridHeight; m++)
{
@@ -111,15 +155,24 @@ public static Jbig2Bitmap DecodeHalftoneRegion(
// a vector, so a dither grid can be rotated or sheared relative to
// the page. Coordinates are in 1/256 pixel units throughout, hence
// the shift rather than a division.
- var x = p.GridX + m * p.VectorY + n * p.VectorX;
- var y = p.GridY + m * p.VectorX - n * p.VectorY;
+ // In long: HGX is a full 32-bit value, and a grid of 2^24 cells times a
+ // 16-bit vector overflows int long before it is implausible.
+ var x = p.GridX + (long)m * p.VectorY + (long)n * p.VectorX;
+ var y = p.GridY + (long)m * p.VectorX - (long)n * p.VectorY;
var level = grey[m * p.GridWidth + n];
if ((uint)level >= (uint)patterns.Length)
throw new InvalidDataException(
$"JBIG2: halftone grey level {level} exceeds the {patterns.Length}-pattern dictionary.");
- region.Combine(patterns[level], x >> 8, y >> 8, p.Combination);
+ var px = x >> 8;
+ var py = y >> 8;
+ // A pattern wholly off the region changes nothing; skipping it here
+ // also keeps the coordinates inside int for Combine.
+ if (px >= p.Width || py >= p.Height || px + patterns[level].Width <= 0 || py + patterns[level].Height <= 0)
+ continue;
+
+ region.Combine(patterns[level], (int)px, (int)py, p.Combination);
}
}
@@ -132,14 +185,21 @@ public static Jbig2Bitmap DecodeHalftoneRegion(
/// above it to undo the Gray coding.
///
private static int[] DecodeGrayscale(
- ref MqDecoder mq, int width, int height, int levels, int template, Jbig2PixelBudget budget)
+ ReadOnlySpan data, HalftoneParameters p, int levels, Jbig2Bitmap? skip, Jbig2PixelBudget budget)
{
+ var width = p.GridWidth;
+ var height = p.GridHeight;
+
+ // HBPP = ceil(log2(HNUMPATS)), at least one plane. levels is at most 2^32,
+ // so this stops by 32.
var planes = 0;
- while (1 << planes < levels) planes++;
+ while (planes < 32 && 1L << planes < levels) planes++;
planes = Math.Max(planes, 1);
- var contexts = new byte[1 << GenericRegionDecoder.ContextBits(template)];
- var at = GrayscaleAt(template);
+ var contexts = new byte[1 << GenericRegionDecoder.ContextBits(p.Template)];
+ var at = GrayscaleAt(p.Template);
+ var mq = new MqDecoder(data);
+ var mmrOffset = 0;
// The caller has already bounded the grid by Jbig2Limits.MaxHalftoneGridCells,
// which is what keeps this product inside int and these two arrays (five
@@ -149,8 +209,19 @@ private static int[] DecodeGrayscale(
for (var bit = planes - 1; bit >= 0; bit--)
{
- var plane = GenericRegionDecoder.Decode(
- ref mq, contexts, width, height, template, typicalPrediction: false, at, budget);
+ Jbig2Bitmap plane;
+ if (p.Mmr)
+ {
+ // C.5 with GSMMR = 1: each plane is its own MMR bitmap, starting on the
+ // byte after the previous one ended. SKIP does not apply to MMR coding.
+ plane = MmrDecoder.Decode(data[Math.Min(mmrOffset, data.Length)..], width, height, budget, out var used);
+ mmrOffset += used;
+ }
+ else
+ {
+ plane = GenericRegionDecoder.Decode(
+ ref mq, contexts, width, height, p.Template, typicalPrediction: false, at, budget, skip);
+ }
// C.5: every plane but the first is coded relative to the one above,
// so the running XOR is what turns Gray code back into binary.
diff --git a/src/Sdcb.SimdCodecs.Jbig2/HuffmanTable.cs b/src/Sdcb.SimdCodecs.Jbig2/HuffmanTable.cs
new file mode 100644
index 0000000..a9829c4
--- /dev/null
+++ b/src/Sdcb.SimdCodecs.Jbig2/HuffmanTable.cs
@@ -0,0 +1,305 @@
+using System;
+using System.Buffers.Binary;
+using System.Collections.Generic;
+using System.IO;
+
+namespace Sdcb.SimdCodecs.Jbig2;
+
+///
+/// A bit reader over Huffman-coded JBIG2 data, most significant bit first
+/// (T.88 §6.2.2, §7.4.13). Unlike , which T.88 lets read
+/// past its data as 0xFF, a Huffman stream has no such convention, so running out
+/// is an error rather than an invented value.
+///
+internal ref struct Jbig2BitReader
+{
+ private readonly ReadOnlySpan _data;
+ private long _bitPosition;
+
+ public Jbig2BitReader(ReadOnlySpan data)
+ {
+ _data = data;
+ _bitPosition = 0;
+ }
+
+ /// The data this reader reads from.
+ public readonly ReadOnlySpan Data => _data;
+
+ /// The byte holding the next unread bit, or the next byte once aligned.
+ public readonly int BytePosition => (int)((_bitPosition + 7) >> 3);
+
+ public int ReadBit()
+ {
+ var index = _bitPosition >> 3;
+ if (index >= _data.Length)
+ throw new InvalidDataException("JBIG2: Huffman-coded data ends early.");
+
+ var bit = (_data[(int)index] >> (7 - (int)(_bitPosition & 7))) & 1;
+ _bitPosition++;
+ return bit;
+ }
+
+ /// Reads (0 to 32) bits as an unsigned value.
+ public uint ReadBits(int count)
+ {
+ if ((uint)count > 32) throw new ArgumentOutOfRangeException(nameof(count));
+ if (_bitPosition + count > (long)_data.Length * 8)
+ throw new InvalidDataException("JBIG2: Huffman-coded data ends early.");
+
+ uint value = 0;
+ for (var i = 0; i < count; i++) value = (value << 1) | (uint)ReadBit();
+ return value;
+ }
+
+ /// Skips to the next byte boundary (a no-op when already on one).
+ public void AlignToByte() => _bitPosition = (_bitPosition + 7) & ~7L;
+
+ /// Moves to byte , which must lie within the data.
+ public void SeekToByte(long position)
+ {
+ if (position < 0 || position > _data.Length)
+ throw new InvalidDataException("JBIG2: Huffman-coded data ends early.");
+ _bitPosition = position * 8;
+ }
+}
+
+///
+/// A Huffman table in the sense of T.88 Annex B: a list of lines, each a prefix
+/// code length (PREFLEN), a range length (RANGELEN) and the bottom of its range
+/// (RANGELOW), plus optionally a lower-range line (values from RANGELOW down to
+/// minus infinity), an upper-range line (RANGELOW up) and an out-of-band line.
+/// Prefix codes are assigned from the lengths alone by the canonical procedure of
+/// B.3, in line order, so a table is fully described by its lines.
+///
+internal sealed class HuffmanTable
+{
+ /// What a line's value means.
+ internal enum LineKind : byte
+ {
+ Normal,
+ Lower,
+ Upper,
+ OutOfBand,
+ }
+
+ /// One table line (T.88 B.2).
+ internal readonly record struct Line(int PrefixLength, int RangeLength, int RangeLow, LineKind Kind = LineKind.Normal);
+
+ /// The value returns for an OOB line; the same sentinel as the arithmetic decoder's.
+ public const int OutOfBand = ArithIntDecoder.OutOfBand;
+
+ ///
+ /// Prefix lengths beyond this are refused. B.2 allows PREFLEN up to 255 in a
+ /// custom table, but a complete code over 2^32 values never needs more than 32.
+ ///
+ private const int MaxPrefixLength = 32;
+
+ // Canonical decoding tables: for each prefix length, the first code of that
+ // length and the lines that have it, in assignment order.
+ private readonly long[] _firstCode;
+ private readonly Line[][] _byLength;
+
+ /// Builds a table from its lines (B.3).
+ /// The lengths do not form a prefix code.
+ public HuffmanTable(IReadOnlyList lines)
+ {
+ var maxLength = 0;
+ foreach (var line in lines)
+ {
+ if (line.PrefixLength is < 0 or > MaxPrefixLength)
+ throw new InvalidDataException($"JBIG2: Huffman table line has prefix length {line.PrefixLength}.");
+ if (line.RangeLength is < 0 or > 32)
+ throw new InvalidDataException($"JBIG2: Huffman table line has range length {line.RangeLength}.");
+ maxLength = Math.Max(maxLength, line.PrefixLength);
+ }
+
+ var counts = new int[maxLength + 1];
+ foreach (var line in lines)
+ if (line.PrefixLength > 0) counts[line.PrefixLength]++;
+
+ _firstCode = new long[maxLength + 1];
+ _byLength = new Line[maxLength + 1][];
+ var lists = new List[maxLength + 1];
+ for (var l = 1; l <= maxLength; l++) lists[l] = new List(counts[l]);
+ foreach (var line in lines)
+ if (line.PrefixLength > 0) lists[line.PrefixLength].Add(line);
+
+ // B.3: FIRSTCODE[len] = (FIRSTCODE[len - 1] + LENCOUNT[len - 1]) * 2, with
+ // LENCOUNT[0] taken as 0. A set of lengths that over-subscribes the code
+ // space (Kraft sum above 1) would hand out a code wider than its length,
+ // which is not a prefix code; that is malformed data, not a table.
+ long first = 0;
+ var previousCount = 0;
+ for (var l = 1; l <= maxLength; l++)
+ {
+ first = (first + previousCount) << 1;
+ if (first + counts[l] > 1L << l)
+ throw new InvalidDataException("JBIG2: Huffman table prefix lengths do not form a prefix code.");
+ _firstCode[l] = first;
+ _byLength[l] = [.. lists[l]];
+ previousCount = counts[l];
+ }
+
+ _byLength[0] = [];
+ }
+
+ /// Decodes one value (B.4), or .
+ public int Decode(ref Jbig2BitReader reader)
+ {
+ long code = 0;
+ for (var length = 1; length < _firstCode.Length; length++)
+ {
+ code = (code << 1) | (uint)reader.ReadBit();
+ var index = code - _firstCode[length];
+ var group = _byLength[length];
+ if (index < 0 || index >= group.Length) continue;
+
+ var line = group[index];
+ switch (line.Kind)
+ {
+ case LineKind.OutOfBand:
+ return OutOfBand;
+ case LineKind.Lower:
+ return Checked(line.RangeLow - (long)reader.ReadBits(line.RangeLength));
+ case LineKind.Upper:
+ return Checked(line.RangeLow + (long)reader.ReadBits(line.RangeLength));
+ default:
+ return Checked(line.RangeLow + (long)reader.ReadBits(line.RangeLength));
+ }
+ }
+
+ throw new InvalidDataException("JBIG2: Huffman code is not in the table.");
+ }
+
+ private static int Checked(long value) => value is >= int.MinValue + 1 and <= int.MaxValue
+ ? (int)value
+ : throw new InvalidDataException("JBIG2: Huffman-decoded value overflows a 32-bit integer.");
+
+ ///
+ /// A table whose lines all have range length zero and values 0, 1, 2, ... in
+ /// order: the shape of the symbol ID tables of §7.4.3.1.7 and of the run code
+ /// table that precedes them.
+ ///
+ public static HuffmanTable FromCodeLengths(ReadOnlySpan lengths)
+ {
+ var lines = new Line[lengths.Length];
+ for (var i = 0; i < lengths.Length; i++) lines[i] = new Line(lengths[i], 0, i);
+ return new HuffmanTable(lines);
+ }
+
+ ///
+ /// Reads a table segment's data part (T.88 §7.4.13, B.2): flags, HTLOW,
+ /// HTHIGH, then the lines packed at HTPS / HTRS bits each.
+ ///
+ public static HuffmanTable ReadSegment(ReadOnlySpan data)
+ {
+ if (data.Length < 9) throw new InvalidDataException("JBIG2: truncated Huffman table segment.");
+
+ var flags = data[0];
+ var hasOob = (flags & 0x01) != 0;
+ var prefixBits = ((flags >> 1) & 0x07) + 1;
+ var rangeBits = ((flags >> 4) & 0x07) + 1;
+ var low = BinaryPrimitives.ReadInt32BigEndian(data[1..]);
+ var high = BinaryPrimitives.ReadInt32BigEndian(data[5..]);
+ if (low >= high)
+ throw new InvalidDataException($"JBIG2: Huffman table segment has HTLOW {low} not below HTHIGH {high}.");
+
+ var reader = new Jbig2BitReader(data[9..]);
+ var lines = new List();
+
+ // Each line costs at least prefixBits + rangeBits >= 2 bits of the segment,
+ // and ReadBits refuses to run past it, so the loop is bounded by the data.
+ long current = low;
+ while (current < high)
+ {
+ var prefix = (int)reader.ReadBits(prefixBits);
+ var range = (int)reader.ReadBits(rangeBits);
+ if (range > 32)
+ throw new InvalidDataException($"JBIG2: Huffman table segment line has range length {range}.");
+ lines.Add(new Line(prefix, range, (int)current));
+ current += 1L << range;
+ }
+
+ lines.Add(new Line((int)reader.ReadBits(prefixBits), 32, low - 1, LineKind.Lower));
+ lines.Add(new Line((int)reader.ReadBits(prefixBits), 32, high, LineKind.Upper));
+ if (hasOob) lines.Add(new Line((int)reader.ReadBits(prefixBits), 0, 0, LineKind.OutOfBand));
+
+ return new HuffmanTable(lines);
+ }
+
+ // ---- the standard tables of Annex B.5 (Tables B.1 to B.15) --------------------
+
+ private static readonly HuffmanTable?[] Standard = new HuffmanTable?[16];
+
+ /// Standard table B., 1 to 15.
+ public static HuffmanTable Get(int number)
+ {
+ lock (Standard)
+ return Standard[number] ??= new HuffmanTable(StandardLines(number));
+ }
+
+ /// The lines of standard table B., in the order the table lists them.
+ internal static Line[] StandardLines(int number) => number switch
+ {
+ 1 => [N(1, 4, 0), N(2, 8, 16), N(3, 16, 272), U(3, 65808)],
+ 2 => [N(1, 0, 0), N(2, 0, 1), N(3, 0, 2), N(4, 3, 3), N(5, 6, 11), U(6, 75), O(6)],
+ 3 => [N(8, 8, -256), N(1, 0, 0), N(2, 0, 1), N(3, 0, 2), N(4, 3, 3), N(5, 6, 11), L(8, -257), U(7, 75), O(6)],
+ 4 => [N(1, 0, 1), N(2, 0, 2), N(3, 0, 3), N(4, 3, 4), N(5, 6, 12), U(5, 76)],
+ 5 => [N(7, 8, -255), N(1, 0, 1), N(2, 0, 2), N(3, 0, 3), N(4, 3, 4), N(5, 6, 12), L(7, -256), U(6, 76)],
+ 6 =>
+ [
+ N(5, 10, -2048), N(4, 9, -1024), N(4, 8, -512), N(4, 7, -256), N(5, 6, -128), N(5, 5, -64), N(4, 5, -32),
+ N(2, 7, 0), N(3, 7, 128), N(3, 8, 256), N(4, 9, 512), N(4, 10, 1024), L(6, -2049), U(6, 2048),
+ ],
+ 7 =>
+ [
+ N(4, 9, -1024), N(3, 8, -512), N(4, 7, -256), N(5, 6, -128), N(5, 5, -64), N(4, 5, -32), N(4, 5, 0),
+ N(5, 5, 32), N(5, 6, 64), N(4, 7, 128), N(3, 8, 256), N(3, 9, 512), N(4, 10, 1024), L(5, -1025), U(5, 2048),
+ ],
+ 8 =>
+ [
+ N(8, 3, -15), N(9, 1, -7), N(8, 1, -5), N(9, 0, -3), N(7, 0, -2), N(4, 0, -1), N(2, 1, 0), N(5, 0, 2),
+ N(6, 0, 3), N(3, 4, 4), N(6, 1, 20), N(4, 4, 22), N(4, 5, 38), N(5, 6, 70), N(5, 7, 134), N(6, 7, 262),
+ N(7, 8, 390), N(6, 10, 646), L(9, -16), U(9, 1670), O(2),
+ ],
+ 9 =>
+ [
+ N(8, 4, -31), N(9, 2, -15), N(8, 2, -11), N(9, 1, -7), N(7, 1, -5), N(4, 1, -3), N(3, 1, -1), N(3, 1, 1),
+ N(5, 1, 3), N(6, 1, 5), N(3, 5, 7), N(6, 2, 39), N(4, 5, 43), N(4, 6, 75), N(5, 7, 139), N(5, 8, 267),
+ N(6, 8, 523), N(7, 9, 779), N(6, 11, 1291), L(9, -32), U(9, 3339), O(2),
+ ],
+ 10 =>
+ [
+ N(7, 4, -21), N(8, 0, -5), N(7, 0, -4), N(5, 0, -3), N(2, 2, -2), N(5, 0, 2), N(6, 0, 3), N(7, 0, 4),
+ N(8, 0, 5), N(2, 6, 6), N(5, 5, 70), N(6, 5, 102), N(6, 6, 134), N(6, 7, 198), N(6, 8, 326),
+ N(6, 9, 582), N(6, 10, 1094), N(7, 11, 2118), L(8, -22), U(8, 4166), O(2),
+ ],
+ 11 =>
+ [
+ N(1, 0, 1), N(2, 1, 2), N(4, 0, 4), N(4, 1, 5), N(5, 1, 7), N(5, 2, 9), N(6, 2, 13), N(7, 2, 17),
+ N(7, 3, 21), N(7, 4, 29), N(7, 5, 45), N(7, 6, 77), U(7, 141),
+ ],
+ 12 =>
+ [
+ N(1, 0, 1), N(2, 0, 2), N(3, 1, 3), N(5, 0, 5), N(5, 1, 6), N(6, 1, 8), N(7, 0, 10), N(7, 1, 11),
+ N(7, 2, 13), N(7, 3, 17), N(7, 4, 25), N(8, 5, 41), U(8, 73),
+ ],
+ 13 =>
+ [
+ N(1, 0, 1), N(3, 0, 2), N(4, 0, 3), N(5, 0, 4), N(4, 1, 5), N(3, 3, 7), N(6, 1, 15), N(6, 2, 17),
+ N(6, 3, 21), N(6, 4, 29), N(6, 5, 45), N(7, 6, 77), U(7, 141),
+ ],
+ 14 => [N(3, 0, -2), N(3, 0, -1), N(1, 0, 0), N(3, 0, 1), N(3, 0, 2)],
+ 15 =>
+ [
+ N(7, 4, -24), N(6, 2, -8), N(5, 1, -4), N(4, 0, -2), N(3, 0, -1), N(1, 0, 0), N(3, 0, 1), N(4, 0, 2),
+ N(5, 1, 3), N(6, 2, 5), N(7, 4, 9), L(7, -25), U(7, 25),
+ ],
+ _ => throw new ArgumentOutOfRangeException(nameof(number), number, "Standard tables are B.1 to B.15."),
+ };
+
+ private static Line N(int prefix, int range, int low) => new(prefix, range, low);
+ private static Line L(int prefix, int low) => new(prefix, 32, low, LineKind.Lower);
+ private static Line U(int prefix, int low) => new(prefix, 32, low, LineKind.Upper);
+ private static Line O(int prefix) => new(prefix, 0, 0, LineKind.OutOfBand);
+}
diff --git a/src/Sdcb.SimdCodecs.Jbig2/Jbig2Decoder.cs b/src/Sdcb.SimdCodecs.Jbig2/Jbig2Decoder.cs
index 3bb8b2a..bd959b4 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/Jbig2Decoder.cs
+++ b/src/Sdcb.SimdCodecs.Jbig2/Jbig2Decoder.cs
@@ -21,23 +21,31 @@ namespace Sdcb.SimdCodecs.Jbig2;
///
///
/// Standalone .jb2 files — which do have a header, and do carry their own
-/// dimensions — go through , and register with the
+/// dimensions — go through , and register with the
/// Sdcb.SimdCodecs facade via .
///
///
-/// Implemented: every region type T.88 defines, arithmetically coded —
-/// generic regions both ways (§6.2.5 templates 0-3 with TPGDON and arbitrary AT
-/// pixels, and MMR / ITU-T T.6 §6.2.6), refinement regions (§6.3), symbol
-/// dictionaries and text regions (§6.4/§6.5, including per-instance refinement
-/// and refine/aggregate symbols), pattern dictionaries and halftone regions
-/// (§6.6/§6.7), page information, and region composition.
-/// Not implemented: the Huffman-coded variants (SDHUFF / SBHUFF and custom
-/// table segments), MMR inside pattern dictionaries and halftone regions,
-/// HENABLESKIP, and TPGRON in a refinement region. Each throws
+/// Implemented: every region type T.88 defines — generic regions (§6.2.5
+/// templates 0-3 with TPGDON and arbitrary AT pixels, MMR / ITU-T T.6 §6.2.6, and
+/// the unknown data length of §7.2.7), refinement regions with or without TPGRON
+/// (§6.3), symbol dictionaries and text regions both arithmetically and
+/// Huffman-coded (§6.4/§6.5, with the standard tables of Annex B, custom table
+/// segments, per-instance refinement and refine/aggregate symbols), pattern
+/// dictionaries and halftone regions with MMR, HENABLESKIP and any GRAYMAX
+/// (§6.6/§6.7), page information, region composition, and every page of a
+/// multi-page file.
+/// Not implemented: EXTTEMPLATE generic regions, symbol dictionaries that
+/// import or retain arithmetic contexts, and T.6's 2D extension codes. Each throws
/// naming the feature — a stream this decoder
/// cannot fully reconstruct fails loudly rather than returning a
/// plausible-looking partial page.
///
+///
+/// After changing this decoder, run Issue1Jbig2CorpusTests against the
+/// files attached to issue #1 (https://github.com/sdcb/SimdCodecs/issues/1): download
+/// and extract them, set SIMDCODECS_ISSUE1_DIR, and every page is compared with
+/// jbig2dec. They are not in the repository because their licence is unknown.
+///
///
public static class Jbig2Decoder
{
@@ -113,16 +121,37 @@ public static Jbig2Image Decode(ReadOnlySpan embedded, int width, int heig
/// A cancellation token to monitor for cancellation requests.
/// The file header or a segment is malformed.
/// The file needs a JBIG2 feature this decoder does not implement yet.
- public static Jbig2Image DecodeFile(ReadOnlySpan file, CancellationToken cancellationToken = default)
+ public static Jbig2Image DecodeFile(ReadOnlySpan file, CancellationToken cancellationToken = default) =>
+ DecodeFile(file, pageNumber: 1, cancellationToken);
+
+ ///
+ /// Decodes one page of a standalone .jb2 file (T.88 Annex D). Segments
+ /// associated with page 0 (shared dictionaries) apply to every page; the rest
+ /// are filtered to .
+ ///
+ /// The standalone JBIG2 file bytes.
+ ///
+ /// The page's association number (T.88 §7.2.6): 1 for the first page. A file's pages
+ /// are numbered 1 to the page count that
+ /// reports.
+ ///
+ /// A cancellation token to monitor for cancellation requests.
+ /// is not positive.
+ /// The file header or a segment is malformed, or the file has no such page.
+ /// The file needs a JBIG2 feature this decoder does not implement yet.
+ public static Jbig2Image DecodeFile(ReadOnlySpan file, int pageNumber, CancellationToken cancellationToken = default)
{
+ ArgumentOutOfRangeException.ThrowIfNegativeOrZero(pageNumber);
+
var segments = ReadFile(file, out var body);
- if (!TryFindPageSize(body, segments, out var width, out var height))
- throw new InvalidDataException("JBIG2: file has no page information segment for page 1, or its height is indeterminate.");
+ if (!TryFindPageSize(body, segments, (uint)pageNumber, out var width, out var height))
+ throw new InvalidDataException(
+ $"JBIG2: file has no page information segment for page {pageNumber}, or its height is indeterminate.");
cancellationToken.ThrowIfCancellationRequested();
var state = new DecodingState(new Jbig2Bitmap(width, height));
- ProcessStream(body, segments, state, targetPage: 1, cancellationToken);
+ ProcessStream(body, segments, state, targetPage: (uint)pageNumber, cancellationToken);
return new Jbig2Image(width, height, state.Page.Data);
}
@@ -138,9 +167,10 @@ public static bool TryReadFileInfo(ReadOnlySpan file, out int width, out i
///
/// , plus the number of pages:
- /// page information segments, one per page (T.88 §7.4.8). Only page 1 is decoded.
+ /// page information segments, one per page (T.88 §7.4.8). The width and height are page 1's;
+ /// decode any page with .
///
- internal static bool TryReadFileInfo(ReadOnlySpan file, out int width, out int height, out int pages)
+ public static bool TryReadFileInfo(ReadOnlySpan file, out int width, out int height, out int pages)
{
width = height = 0;
pages = 0;
@@ -151,7 +181,7 @@ internal static bool TryReadFileInfo(ReadOnlySpan file, out int width, out
{
if (segment.Type == SegmentType.PageInformation) pages++;
}
- return TryFindPageSize(body, segments, out width, out height);
+ return TryFindPageSize(body, segments, 1, out width, out height);
}
catch (Exception e) when (e is InvalidDataException or NotSupportedException or ArgumentException)
{
@@ -193,6 +223,7 @@ private static List ReadSequential(ReadOnlySpan stream)
while (position < stream.Length)
{
var header = Jbig2Segment.ReadHeader(stream, ref position);
+ if (header.LengthUnknown) header = Jbig2Segment.ResolveUnknownLength(stream, header);
segments.Add(header);
position = header.DataStart + header.DataLength;
}
@@ -216,6 +247,10 @@ private static List ReadRandomAccess(ReadOnlySpan stream)
while (position + declared < stream.Length)
{
var header = Jbig2Segment.ReadHeader(stream, ref position);
+ if (header.LengthUnknown)
+ throw new InvalidDataException(
+ "JBIG2: a random-access file has a segment of unknown data length, which T.88 §7.2.7 " +
+ "allows only where each data part follows its header.");
headers.Add(header);
declared += header.DataLength;
}
@@ -236,13 +271,17 @@ private static List ReadRandomAccess(ReadOnlySpan stream)
// ---- page geometry -----------------------------------------------------------
- private static bool TryFindPageSize(ReadOnlySpan stream, List segments, out int width, out int height)
+ private static bool TryFindPageSize(
+ ReadOnlySpan stream, List segments, uint page, out int width, out int height)
{
width = height = 0;
foreach (var segment in segments)
{
- if (segment.Type != SegmentType.PageInformation || segment.Page is not (0 or 1)) continue;
+ // Page 1 also accepts a page information segment on page 0, which some
+ // single-page encoders write.
+ if (segment.Type != SegmentType.PageInformation
+ || !(segment.Page == page || (page == 1 && segment.Page == 0))) continue;
if (segment.DataLength < 19) return false;
var info = stream.Slice(segment.DataStart, segment.DataLength);
@@ -251,7 +290,7 @@ private static bool TryFindPageSize(ReadOnlySpan stream, List int.MaxValue || h > int.MaxValue
@@ -266,14 +305,15 @@ private static bool TryFindPageSize(ReadOnlySpan stream, List stream, List segments, out uint height)
+ private static bool TryFindStripedHeight(
+ ReadOnlySpan stream, List segments, uint page, out uint height)
{
// §7.4.10: an end-of-stripe segment carries the row number of the stripe's
// last row, so the tallest one plus one is the page height.
long last = -1;
foreach (var segment in segments)
{
- if (segment.Type != SegmentType.EndOfStripe || segment.DataLength < 4) continue;
+ if (segment.Type != SegmentType.EndOfStripe || segment.DataLength < 4 || segment.Page != page) continue;
last = Math.Max(last, BinaryPrimitives.ReadUInt32BigEndian(stream.Slice(segment.DataStart, 4)));
}
@@ -301,6 +341,7 @@ private sealed class DecodingState(Jbig2Bitmap page)
private readonly Dictionary _intermediateRegions = [];
private readonly Dictionary _symbolDictionaries = [];
private readonly Dictionary _patternDictionaries = [];
+ private readonly Dictionary _tables = [];
/// The page being composited onto.
public Jbig2Bitmap Page { get; } = page;
@@ -324,6 +365,24 @@ public void StoreIntermediateRegion(uint segment, Jbig2Bitmap bitmap) =>
public void StoreSymbols(uint segment, Jbig2Bitmap[] symbols) =>
_symbolDictionaries[segment] = symbols;
+ /// Records a custom Huffman table (§7.4.13).
+ public void StoreTable(uint segment, HuffmanTable table) => _tables[segment] = table;
+
+ ///
+ /// The custom Huffman tables a segment refers to, in referred-to order: the
+ /// order in which its table selection flags consume them (§7.4.2.1.6,
+ /// §7.4.3.1.6).
+ ///
+ public Queue GatherTables(SegmentHeader segment)
+ {
+ var tables = new Queue();
+ foreach (var referred in segment.ReferredTo)
+ if (_tables.TryGetValue(referred, out var table))
+ tables.Enqueue(table);
+
+ return tables;
+ }
+
/// Records a pattern dictionary's dither patterns.
public void StorePatterns(uint segment, Jbig2Bitmap[] patterns) =>
_patternDictionaries[segment] = patterns;
@@ -417,7 +476,7 @@ private static void ProcessSegment(SegmentHeader segment, ReadOnlySpan dat
case SegmentType.ImmediateGenericRegion:
case SegmentType.ImmediateLosslessGenericRegion:
- Compose(state, DecodeGenericRegion(data, state.Budget, out var genericInfo), genericInfo);
+ Compose(state, DecodeGenericRegion(data, state.Budget, out var genericInfo, segment.RowCount), genericInfo);
break;
case SegmentType.IntermediateGenericRegion:
@@ -457,9 +516,8 @@ private static void ProcessSegment(SegmentHeader segment, ReadOnlySpan dat
break;
case SegmentType.Tables:
- throw new NotSupportedException(
- "JBIG2: custom Huffman table segments are not implemented — this decoder handles " +
- "arithmetic symbol dictionaries and text regions only.");
+ state.StoreTable(segment.Number, HuffmanTable.ReadSegment(data));
+ break;
case SegmentType.PatternDictionary:
state.StorePatterns(segment.Number, DecodePatternDictionary(data, state.Budget));
@@ -511,10 +569,10 @@ private static void ApplyPageInformation(ReadOnlySpan data, Jbig2Bitmap pa
/// Generic region segment (T.88 §7.4.6): region info, flags, AT pixels, then MQ-coded data.
private static Jbig2Bitmap DecodeGenericRegion(
- ReadOnlySpan data, Jbig2PixelBudget budget, out RegionInfo info)
+ ReadOnlySpan data, Jbig2PixelBudget budget, out RegionInfo info, uint? rowCount = null)
{
var position = 0;
- var region = Jbig2Segment.ReadRegionInfo(data, ref position);
+ var region = Jbig2Segment.ReadRegionInfo(data, ref position, rowCount);
info = region;
if (position >= data.Length)
@@ -644,10 +702,6 @@ private static Jbig2Bitmap[] DecodeSymbolDictionary(
var contextUsed = (flags & 0x0100) != 0;
var contextRetained = (flags & 0x0200) != 0;
- if (huffman)
- throw new NotSupportedException(
- "JBIG2: Huffman-coded symbol dictionaries (SDHUFF = 1) are not implemented.");
-
// §7.4.3.1.1 bits 8-9: carrying adaptive state across segments. Refusing
// is not laziness — a dictionary decoded with the wrong initial contexts
// produces plausible but wrong glyphs, which is precisely the failure this
@@ -656,12 +710,16 @@ private static Jbig2Bitmap[] DecodeSymbolDictionary(
throw new NotSupportedException(
"JBIG2: symbol dictionaries that import or retain arithmetic contexts are not implemented.");
+ // §7.4.2.1.2: the generic AT pixels are present only for arithmetic coding.
Span at = stackalloc sbyte[8];
- var atPixels = template == 0 ? 4 : 1;
- if (position + atPixels * 2 > data.Length)
- throw new InvalidDataException("JBIG2: truncated symbol dictionary AT pixel list.");
+ if (!huffman)
+ {
+ var atPixels = template == 0 ? 4 : 1;
+ if (position + atPixels * 2 > data.Length)
+ throw new InvalidDataException("JBIG2: truncated symbol dictionary AT pixel list.");
- for (var i = 0; i < atPixels * 2; i++) at[i] = (sbyte)data[position++];
+ for (var i = 0; i < atPixels * 2; i++) at[i] = (sbyte)data[position++];
+ }
Span refinementAt = stackalloc sbyte[4];
RefinementRegionDecoder.NominalAt.CopyTo(refinementAt);
@@ -694,6 +752,21 @@ private static Jbig2Bitmap[] DecodeSymbolDictionary(
var parameters = new SymbolDictionaryParameters(
(int)created, (int)exported, template, refinementAggregate, refinementTemplate);
+ if (huffman)
+ {
+ // §7.4.2.1.1 bits 2-7: which table each number is read with, standard or
+ // the next custom one this segment refers to.
+ var custom = state.GatherTables(segment);
+ var tables = new SymbolDictionaryDecoder.HuffmanTables(
+ SelectTable((flags >> 2) & 0x03, [4, 5, 0], custom, "SDHUFFDH"),
+ SelectTable((flags >> 4) & 0x03, [2, 3, 0], custom, "SDHUFFDW"),
+ SelectTable((flags >> 6) & 0x01, [1], custom, "SDHUFFBMSIZE"),
+ SelectTable((flags >> 7) & 0x01, [1], custom, "SDHUFFAGGINST"));
+
+ return SymbolDictionaryDecoder.DecodeHuffman(
+ data[position..], parameters, inputSymbols, tables, refinementAt, state.Budget);
+ }
+
var mq = new MqDecoder(data[position..]);
return SymbolDictionaryDecoder.Decode(
ref mq, parameters, inputSymbols, at, refinementAt, state.Budget);
@@ -729,9 +802,15 @@ private static Jbig2Bitmap DecodeTextRegion(
var dsOffset = (flags >> 10) & 0x1F;
if (dsOffset > 15) dsOffset -= 32;
+ // §7.4.3.1.2: the Huffman table selections come next, when SBHUFF is set.
+ var huffmanFlags = 0;
if (huffman)
- throw new NotSupportedException(
- "JBIG2: Huffman-coded text regions (SBHUFF = 1) are not implemented.");
+ {
+ if (position + 2 > data.Length)
+ throw new InvalidDataException("JBIG2: truncated text region Huffman flags.");
+ huffmanFlags = BinaryPrimitives.ReadUInt16BigEndian(data[position..]);
+ position += 2;
+ }
Span refinementAt = stackalloc sbyte[4];
RefinementRegionDecoder.NominalAt.CopyTo(refinementAt);
@@ -762,18 +841,59 @@ private static Jbig2Bitmap DecodeTextRegion(
region.Width, region.Height, (int)instances, 1 << logStrips,
defaultPixel, combination, corner, transposed, dsOffset, refine, refinementTemplate);
- var codeLength = TextRegionDecoder.SymbolCodeLength(symbols.Length);
- var idContexts = new byte[ArithIntDecoder.IdContextSize(codeLength)];
var refinementContexts = refine
? new byte[1 << RefinementRegionDecoder.ContextBits(refinementTemplate)]
: [];
+ if (huffman)
+ {
+ var custom = state.GatherTables(segment);
+ var tables = new TextRegionDecoder.HuffmanTables(
+ SelectTable(huffmanFlags & 0x03, [6, 7, 0], custom, "SBHUFFFS"),
+ SelectTable((huffmanFlags >> 2) & 0x03, [8, 9, 10], custom, "SBHUFFDS"),
+ SelectTable((huffmanFlags >> 4) & 0x03, [11, 12, 13], custom, "SBHUFFDT"),
+ SelectTable((huffmanFlags >> 6) & 0x03, [14, 15, 0], custom, "SBHUFFRDW"),
+ SelectTable((huffmanFlags >> 8) & 0x03, [14, 15, 0], custom, "SBHUFFRDH"),
+ SelectTable((huffmanFlags >> 10) & 0x03, [14, 15, 0], custom, "SBHUFFRDX"),
+ SelectTable((huffmanFlags >> 12) & 0x03, [14, 15, 0], custom, "SBHUFFRDY"),
+ SelectTable((huffmanFlags >> 14) & 0x01, [1], custom, "SBHUFFRSIZE"));
+
+ var reader = new Jbig2BitReader(data[position..]);
+ var symbolCodes = TextRegionDecoder.ReadSymbolCodeTable(ref reader, symbols.Length);
+ return TextRegionDecoder.DecodeHuffman(
+ ref reader, parameters, symbols, tables, symbolCodes, codeLength: 0,
+ refinementContexts, refinementAt, state.Budget);
+ }
+
+ var codeLength = TextRegionDecoder.SymbolCodeLength(symbols.Length);
+ var idContexts = new byte[ArithIntDecoder.IdContextSize(codeLength)];
+
var mq = new MqDecoder(data[position..]);
return TextRegionDecoder.Decode(
ref mq, parameters, symbols, new TextRegionDecoder.Fields(),
idContexts, refinementContexts, refinementAt, state.Budget);
}
+ ///
+ /// Resolves one Huffman table selection field. maps
+ /// each value to a standard table number, 0 marking a reserved value; the value
+ /// one past the list (3 for a two-bit field, 1 for a one-bit field) means "the
+ /// next custom table".
+ ///
+ private static HuffmanTable SelectTable(int selection, int[] standard, Queue custom, string field)
+ {
+ var customValue = standard.Length == 1 ? 1 : 3;
+ if (selection == customValue)
+ return custom.Count > 0
+ ? custom.Dequeue()
+ : throw new InvalidDataException($"JBIG2: {field} selects a custom Huffman table the segment does not refer to.");
+
+ var number = selection < standard.Length ? standard[selection] : 0;
+ return number != 0
+ ? HuffmanTable.Get(number)
+ : throw new InvalidDataException($"JBIG2: {field} selection {selection} is reserved.");
+ }
+
///
/// Pattern dictionary segment (T.88 §7.4.4): flags, the pattern size, the
/// highest grey level, then one wide collective bitmap holding every pattern.
@@ -791,15 +911,23 @@ private static Jbig2Bitmap[] DecodePatternDictionary(ReadOnlySpan data, Jb
int patternHeight = data[2];
var maxIndex = BinaryPrimitives.ReadUInt32BigEndian(data[3..]);
- if (mmr)
- throw new NotSupportedException("JBIG2: MMR-coded pattern dictionaries are not implemented.");
if (patternWidth == 0 || patternHeight == 0)
throw new InvalidDataException($"JBIG2: pattern dictionary has a {patternWidth}x{patternHeight} pattern size.");
- // §7.4.4.1.4 caps GRAYMAX at 255, and the collective bitmap is
- // (GRAYMAX + 1) patterns wide, so this also bounds the allocation.
- if (maxIndex > 255)
- throw new InvalidDataException($"JBIG2: pattern dictionary GRAYMAX {maxIndex} exceeds 255.");
+ // GRAYMAX is a four-byte field (§7.4.5.1.4), and real dictionaries go past
+ // 255 (a 20x20 cell has 401 grey levels). The collective bitmap is
+ // (GRAYMAX + 1) patterns wide, so bounding that one bitmap bounds the
+ // allocation; the pixel budget is charged by the region decoder as usual.
+ var collectiveWidth = (maxIndex + 1L) * patternWidth;
+ if (collectiveWidth > int.MaxValue || collectiveWidth * patternHeight > Jbig2Limits.MaxBitmapPixels)
+ throw new InvalidDataException(
+ $"JBIG2: pattern dictionary GRAYMAX {maxIndex} with {patternWidth}x{patternHeight} patterns " +
+ $"needs a {collectiveWidth}x{patternHeight} collective bitmap, past the " +
+ $"{Jbig2Limits.MaxBitmapPixels:N0} pixel ceiling.");
+
+ if (mmr)
+ return HalftoneDecoder.DecodePatternDictionaryMmr(
+ data[7..], patternWidth, patternHeight, (int)maxIndex, budget);
var mq = new MqDecoder(data[7..]);
return HalftoneDecoder.DecodePatternDictionary(
@@ -828,14 +956,9 @@ private static Jbig2Bitmap DecodeHalftoneRegion(
var combination = (CombinationOperator)((flags >> 4) & 0x07);
var defaultPixel = (byte)((flags >> 7) & 0x01);
- if (mmr)
- throw new NotSupportedException("JBIG2: MMR-coded halftone regions are not implemented.");
-
- // HENABLESKIP changes what gets coded, not just how fast: skipped pixels
- // are absent from the stream entirely, so ignoring the flag would
- // desynchronise rather than merely run slower.
- if (enableSkip)
- throw new NotSupportedException("JBIG2: halftone regions using HENABLESKIP are not implemented.");
+ // HENABLESKIP changes what gets coded, not just how fast: skipped cells are
+ // absent from the stream entirely, which is why HalftoneDecoder rebuilds
+ // the same SKIP bitmap the encoder used before reading any plane.
if (combination > CombinationOperator.Replace)
throw new InvalidDataException($"JBIG2: reserved halftone combination operator {(int)combination}.");
@@ -858,10 +981,9 @@ private static Jbig2Bitmap DecodeHalftoneRegion(
var parameters = new HalftoneDecoder.HalftoneParameters(
region.Width, region.Height, (int)gridWidth, (int)gridHeight,
- gridX, gridY, vectorX, vectorY, template, defaultPixel, combination);
+ gridX, gridY, vectorX, vectorY, template, defaultPixel, combination, mmr, enableSkip);
- var mq = new MqDecoder(data[position..]);
- return HalftoneDecoder.DecodeHalftoneRegion(ref mq, parameters, patterns, state.Budget);
+ return HalftoneDecoder.DecodeHalftoneRegion(data[position..], parameters, patterns, state.Budget);
}
}
diff --git a/src/Sdcb.SimdCodecs.Jbig2/Jbig2ImageDecoder.cs b/src/Sdcb.SimdCodecs.Jbig2/Jbig2ImageDecoder.cs
index 2b466cd..52535d7 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/Jbig2ImageDecoder.cs
+++ b/src/Sdcb.SimdCodecs.Jbig2/Jbig2ImageDecoder.cs
@@ -9,7 +9,7 @@ namespace Sdcb.SimdCodecs.Jbig2;
///
/// adapter for standalone .jb2 files, bridging
-/// into the Sdcb.SimdCodecs facade.
+/// into the Sdcb.SimdCodecs facade.
///
/// This is the courtesy entry point, not the main one. JBIG2's reason to
/// exist in this family is PDF's /JBIG2Decode filter, and a PDF-embedded
diff --git a/src/Sdcb.SimdCodecs.Jbig2/Jbig2Segment.cs b/src/Sdcb.SimdCodecs.Jbig2/Jbig2Segment.cs
index e856469..d9edc26 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/Jbig2Segment.cs
+++ b/src/Sdcb.SimdCodecs.Jbig2/Jbig2Segment.cs
@@ -47,15 +47,28 @@ internal readonly record struct SegmentHeader(
uint[] ReferredTo,
uint Page,
int DataStart,
- int DataLength);
+ int DataLength)
+{
+ ///
+ /// For an immediate generic region whose header declared an unknown data length
+ /// (T.88 §7.2.7), the row count that follows its end sequence, which replaces the
+ /// region's declared height (§7.4.6.4). Null for every other segment.
+ ///
+ public uint? RowCount { get; init; }
+
+ /// Whether the header declared 0xFFFFFFFF and the length is still to be found.
+ public bool LengthUnknown => DataLength < 0;
+}
/// Parsing of T.88 segment headers and the segment-level fields they introduce.
internal static class Jbig2Segment
{
///
/// T.88 §7.2.7 unknown-length marker. Legal only on an immediate generic
- /// region, where the data part is instead terminated by a search sequence;
- /// this decoder rejects it rather than guessing.
+ /// region in a sequential or embedded stream, where the data part is instead
+ /// terminated by an end sequence and a row count.
+ /// reports it as a negative , and the
+ /// sequential reader resolves it with .
///
public const uint UnknownDataLength = 0xFFFFFFFF;
@@ -114,25 +127,74 @@ public static SegmentHeader ReadHeader(ReadOnlySpan data, ref int position
var length = ReadUInt32(data, ref position);
if (length == UnknownDataLength)
- throw new NotSupportedException(
- "JBIG2: segment with unknown data length (0xFFFFFFFF, T.88 §7.2.7) is not supported.");
+ {
+ if (type is not (SegmentType.ImmediateGenericRegion or SegmentType.ImmediateLosslessGenericRegion))
+ throw new InvalidDataException(
+ $"JBIG2: segment {number} of type {(int)type} declares an unknown data length, which " +
+ "T.88 §7.2.7 allows only for an immediate generic region.");
+ return new SegmentHeader(number, type, referredTo, page, position, -1);
+ }
if (length > int.MaxValue || position + (long)length > data.Length)
throw new InvalidDataException($"JBIG2: segment {number} data length {length} runs past the end of the stream.");
return new SegmentHeader(number, type, referredTo, page, position, (int)length);
}
+ ///
+ /// Finds the end of an immediate generic region whose data length is unknown
+ /// (T.88 §7.2.7, §7.4.6.4). After the 18 bytes of region info and flags (and the
+ /// AT pixels, for arithmetic coding) the coded data runs until an end sequence,
+ /// 00 00 for MMR and FF AC for arithmetic coding, followed by a
+ /// four-byte row count. Neither sequence can occur inside its own kind of coded
+ /// data: MQ output never has FF followed by a byte above 8F, and
+ /// T.6 codes never hold sixteen zero bits in a row. The resolved data length
+ /// covers the sequence and the row count.
+ ///
+ public static SegmentHeader ResolveUnknownLength(ReadOnlySpan stream, SegmentHeader header)
+ {
+ var start = header.DataStart;
+ if (start + 18 > stream.Length)
+ throw new InvalidDataException("JBIG2: truncated generic region of unknown length.");
+
+ var flags = stream[start + 17];
+ var mmr = (flags & 0x01) != 0;
+ var codedStart = start + 18;
+ if (!mmr) codedStart += ((flags >> 1) & 0x03) == 0 ? 8 : 2;
+
+ if (codedStart > stream.Length)
+ throw new InvalidDataException("JBIG2: truncated generic region of unknown length.");
+
+ var (first, second) = mmr ? ((byte)0x00, (byte)0x00) : ((byte)0xFF, (byte)0xAC);
+ var at = stream[codedStart..].IndexOf([first, second]);
+ // The search is linear in the stream, once per segment, so a hostile stream
+ // costs no more than reading it.
+ if (at < 0 || codedStart + at + 6 > stream.Length)
+ throw new InvalidDataException(
+ $"JBIG2: generic region segment {header.Number} has an unknown data length and no end sequence.");
+
+ var end = codedStart + at;
+ var rows = BinaryPrimitives.ReadUInt32BigEndian(stream[(end + 2)..]);
+ return header with { DataLength = end + 6 - start, RowCount = rows };
+ }
+
///
/// The region segment information field (T.88 §7.4.1): 17 bytes of geometry
/// and the external combination operator, prefixed to every region segment.
///
- public static RegionInfo ReadRegionInfo(ReadOnlySpan data, ref int position)
+ /// The segment data.
+ /// Where the field starts; advanced past it.
+ ///
+ /// The row count of an unknown-length generic region, which replaces the declared
+ /// height (§7.4.6.4), or null.
+ ///
+ public static RegionInfo ReadRegionInfo(ReadOnlySpan data, ref int position, uint? rowCount = null)
{
if (position + 17 > data.Length)
throw new InvalidDataException("JBIG2: truncated region segment information field.");
var width = ReadUInt32(data, ref position);
var height = ReadUInt32(data, ref position);
+ if (rowCount is { } rows) height = rows;
var x = ReadUInt32(data, ref position);
var y = ReadUInt32(data, ref position);
var flags = ReadByte(data, ref position);
diff --git a/src/Sdcb.SimdCodecs.Jbig2/MmrDecoder.cs b/src/Sdcb.SimdCodecs.Jbig2/MmrDecoder.cs
index 2d0e99e..03c37b3 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/MmrDecoder.cs
+++ b/src/Sdcb.SimdCodecs.Jbig2/MmrDecoder.cs
@@ -49,7 +49,17 @@ private MmrDecoder(ReadOnlySpan data)
///
/// The coded data is malformed or ends early.
/// The data uses a 2D extension code, which includes T.6's uncompressed mode.
- public static Jbig2Bitmap Decode(ReadOnlySpan data, int width, int height, Jbig2PixelBudget budget)
+ public static Jbig2Bitmap Decode(ReadOnlySpan data, int width, int height, Jbig2PixelBudget budget) =>
+ Decode(data, width, height, budget, out _);
+
+ ///
+ /// , also reporting how many
+ /// bytes the bitmap occupied: its rows, an EOFB if one follows them, and the padding to the
+ /// next byte boundary. That is where the next MMR bitmap starts when several are coded back to
+ /// back, as the bitplanes of a halftone region's grey-scale image are (T.88 Annex C.5).
+ ///
+ public static Jbig2Bitmap Decode(
+ ReadOnlySpan data, int width, int height, Jbig2PixelBudget budget, out int bytesConsumed)
{
budget.Charge(width, height);
var bitmap = new Jbig2Bitmap(width, height);
@@ -75,6 +85,10 @@ public static Jbig2Bitmap Decode(ReadOnlySpan data, int width, int height,
referenceCount = count;
}
+ // T.6 §2.4: EOFB is two EOL codes, 000000000001 twice. It is optional after a
+ // region of known height, so it is consumed only when it is actually there.
+ if (decoder.Peek(24) == 0x001001) decoder._bitPosition += 24;
+ bytesConsumed = Math.Min((decoder._bitPosition + 7) >> 3, data.Length);
return bitmap;
}
@@ -246,7 +260,7 @@ private readonly uint Peek(int count)
for (var i = 0; i < 4; i++)
window = (window << 8) | (index + i < _data.Length ? _data[index + i] : 0u);
- // count is at most 13 and shift at most 7, so the wanted bits are always
+ // count is at most 24 and shift at most 7, so the wanted bits are always
// inside the 32-bit window.
return (window << shift) >> (32 - count);
}
diff --git a/src/Sdcb.SimdCodecs.Jbig2/PublicAPI.Unshipped.txt b/src/Sdcb.SimdCodecs.Jbig2/PublicAPI.Unshipped.txt
index 7dc5c58..4395a9c 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/PublicAPI.Unshipped.txt
+++ b/src/Sdcb.SimdCodecs.Jbig2/PublicAPI.Unshipped.txt
@@ -1 +1,3 @@
#nullable enable
+static Sdcb.SimdCodecs.Jbig2.Jbig2Decoder.DecodeFile(System.ReadOnlySpan file, int pageNumber, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken)) -> Sdcb.SimdCodecs.Jbig2.Jbig2Image!
+static Sdcb.SimdCodecs.Jbig2.Jbig2Decoder.TryReadFileInfo(System.ReadOnlySpan file, out int width, out int height, out int pages) -> bool
diff --git a/src/Sdcb.SimdCodecs.Jbig2/RefinementRegionDecoder.cs b/src/Sdcb.SimdCodecs.Jbig2/RefinementRegionDecoder.cs
index c8b4533..c736d9e 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/RefinementRegionDecoder.cs
+++ b/src/Sdcb.SimdCodecs.Jbig2/RefinementRegionDecoder.cs
@@ -25,26 +25,19 @@ namespace Sdcb.SimdCodecs.Jbig2;
/// contexts that name a specific neighbourhood in the spec's numbering.
///
///
-/// TPGRON is refused. Everything else here is confirmed against jbig2dec
-/// — both templates, six reference/target relationships, and moved AT pixels —
-/// but with TPGRON on, this decoder and jbig2dec part company. What is known,
-/// from measurement rather than assumption:
-///
-///
-/// - GRTEMPLATE 1 never agrees. Sweeping all 1024 possible SLTP
-/// context values produced no agreeing candidate, so the SLTP constant is not
-/// the cause — something about how the skip itself applies is.
-/// - GRTEMPLATE 0 agrees on five of six relationships and desynchronises
-/// part-way down the sixth, which rules out a simple constant just as
-/// firmly.
-/// - jbig2dec emits no diagnostic either way, so there is no evidence about
-/// which side is wrong — and no second oracle available to break the tie.
-///
-///
-/// So the flag throws. Partial credit is the wrong instinct here: "agrees on
-/// five of six" describes a decoder that silently corrupts the sixth, and a
-/// caller cannot tell which one it has. TPGRON is an encoder's option, never a
-/// requirement, so refusing it costs strictly less than getting it wrong.
+/// TPGRON (§6.3.5.6) toggles LTP with one SLTP decision per row, and
+/// while LTP is on a pixel over a uniform 3x3 patch of the reference takes the
+/// patch's colour without a decision. The SLTP decision has a context of its
+/// own, a fixed value: 0x0010 for GRTEMPLATE 0 and 0x0004 for GRTEMPLATE 1 in
+/// the numbering below. An earlier version of this file used 0x0100 / 0x0080
+/// and refused TPGRON because it disagreed with jbig2dec. The constants were
+/// settled by measurement: for GRTEMPLATE 1, sweeping all 1024 values against
+/// jbig2dec over the six Jbig2RefinementTests.Cases leaves 0x0004 as the
+/// only one that agrees on every case; for GRTEMPLATE 0 those cases agree with
+/// many values (the slot is rarely reached), and 0x0010 is the one that also
+/// decodes the TPGRON refinement region of the issue #1 corpus
+/// (042_24.jb2, see Issue1Jbig2CorpusTests) exactly as
+/// jbig2dec does.
///
///
internal static class RefinementRegionDecoder
@@ -57,7 +50,7 @@ internal static class RefinementRegionDecoder
public static ReadOnlySpan NominalAt => [-1, -1, -1, -1];
// T.88 §6.3.5.6: the pseudo-context the SLTP decision uses when TPGRON is on.
- private static ReadOnlySpan TypicalPredictionContexts => [0x0100, 0x0080];
+ private static ReadOnlySpan TypicalPredictionContexts => [0x0010, 0x0004];
/// Number of context bits (hence the context array size) for each GRTEMPLATE.
public static int ContextBits(int template) => template switch
@@ -102,14 +95,6 @@ public static Jbig2Bitmap Decode(
if (contexts.Length < 1 << ContextBits(template))
throw new ArgumentException("Context array too small for this GRTEMPLATE.", nameof(contexts));
- // See the class remarks. Refinement without TPGRON is confirmed against
- // jbig2dec on both templates; with it, this decoder disagrees on inputs it
- // cannot explain, so the flag is refused rather than guessed at.
- if (typicalPrediction)
- throw new NotSupportedException(
- "JBIG2: TPGRON (typical prediction) in a refinement region is not implemented — the decoded " +
- "result cannot be reconciled with the reference decoder, and a wrong page is worse than none.");
-
budget.Charge(width, height);
var bitmap = new Jbig2Bitmap(width, height);
var sltpContext = TypicalPredictionContexts[template];
diff --git a/src/Sdcb.SimdCodecs.Jbig2/SymbolDictionaryDecoder.cs b/src/Sdcb.SimdCodecs.Jbig2/SymbolDictionaryDecoder.cs
index 71ad2ae..be16533 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/SymbolDictionaryDecoder.cs
+++ b/src/Sdcb.SimdCodecs.Jbig2/SymbolDictionaryDecoder.cs
@@ -33,8 +33,10 @@ internal readonly record struct SymbolDictionaryParameters(
/// every page of a scanned document.
///
///
-/// Huffman-coded dictionaries (SDHUFF = 1) are not implemented; the caller
-/// rejects them before reaching here.
+/// Huffman-coded dictionaries (SDHUFF = 1) go through :
+/// the same height classes, but the numbers come from Huffman tables, and unless
+/// SDREFAGG is set each height class's bitmaps arrive together as one collective
+/// bitmap, uncompressed or MMR-coded (§6.5.9).
///
///
internal static class SymbolDictionaryDecoder
@@ -133,7 +135,224 @@ public static Jbig2Bitmap[] Decode(
$"JBIG2: symbol dictionary height class at height {height} codes no symbols.");
}
- return SelectExported(ref mq, ex, inputSymbols, newSymbols, p.ExportedSymbols);
+ var selector = new ExportSelector(inputSymbols, newSymbols, p.ExportedSymbols);
+ while (!selector.Done) selector.Take(ex.Decode(ref mq));
+ return selector.Result();
+ }
+
+ /// The Huffman tables a Huffman-coded dictionary reads its numbers with (§7.4.2.1.1).
+ internal sealed record HuffmanTables(
+ HuffmanTable DeltaHeight, HuffmanTable DeltaWidth, HuffmanTable BitmapSize, HuffmanTable AggregateInstances);
+
+ ///
+ /// Decodes a Huffman-coded dictionary (SDHUFF = 1, T.88 §6.5.5 with the Huffman
+ /// branches of §6.5.6 to §6.5.10).
+ ///
+ /// The coded data, after the segment's fixed fields.
+ /// Parameters from the segment header.
+ /// SDINSYMS.
+ /// SDHUFFDH, SDHUFFDW, SDHUFFBMSIZE and SDHUFFAGGINST.
+ /// SDRAT, used only when SDREFAGG is set.
+ /// The decode's remaining pixel allowance.
+ public static Jbig2Bitmap[] DecodeHuffman(
+ ReadOnlySpan data,
+ SymbolDictionaryParameters p,
+ Jbig2Bitmap[] inputSymbols,
+ HuffmanTables tables,
+ scoped ReadOnlySpan refinementAt,
+ Jbig2PixelBudget budget)
+ {
+ var reader = new Jbig2BitReader(data);
+ var refinementContexts = p.RefinementAggregate
+ ? new byte[1 << RefinementRegionDecoder.ContextBits(p.RefinementTemplate)]
+ : [];
+
+ // §6.5.8.2.3, Huffman case: the symbol IDs of a refinement or aggregate are
+ // plain binary numbers SBSYMCODELEN = ceil(log2(SDNUMINSYMS + SDNUMNEWSYMS))
+ // bits wide.
+ var total = inputSymbols.Length + p.NewSymbols;
+ var codeLength = 0;
+ while (1L << codeLength < total) codeLength++;
+
+ var newSymbols = new List(Math.Min(p.NewSymbols, 1 << 16));
+ var widths = new List();
+ var height = 0;
+
+ while (newSymbols.Count < p.NewSymbols)
+ {
+ var deltaHeight = tables.DeltaHeight.Decode(ref reader);
+ if (deltaHeight == HuffmanTable.OutOfBand)
+ throw new InvalidDataException("JBIG2: unexpected OOB decoding a symbol dictionary height class.");
+
+ height = (int)Math.Clamp((long)height + deltaHeight, int.MinValue, int.MaxValue);
+ if (height <= 0)
+ throw new InvalidDataException($"JBIG2: symbol dictionary height class has non-positive height {height}.");
+
+ var countBeforeClass = newSymbols.Count;
+ var width = 0;
+ long totalWidth = 0;
+ widths.Clear();
+
+ while (true)
+ {
+ var deltaWidth = tables.DeltaWidth.Decode(ref reader);
+ if (deltaWidth == HuffmanTable.OutOfBand) break;
+
+ width = (int)Math.Clamp((long)width + deltaWidth, int.MinValue, int.MaxValue);
+ if (width <= 0)
+ throw new InvalidDataException($"JBIG2: symbol dictionary symbol has non-positive width {width}.");
+ if (newSymbols.Count + widths.Count >= p.NewSymbols)
+ throw new InvalidDataException("JBIG2: symbol dictionary codes more symbols than it declared.");
+
+ if (p.RefinementAggregate)
+ {
+ newSymbols.Add(DecodeAggregateHuffman(
+ ref reader, p, inputSymbols, newSymbols, width, height, codeLength,
+ tables, refinementContexts, refinementAt, budget));
+ }
+ else
+ {
+ widths.Add(width);
+ totalWidth += width;
+ }
+ }
+
+ if (!p.RefinementAggregate && widths.Count > 0)
+ DecodeCollective(ref reader, tables, widths, totalWidth, height, newSymbols, budget);
+
+ // The same progress guard as the arithmetic path.
+ if (newSymbols.Count == countBeforeClass)
+ throw new InvalidDataException(
+ $"JBIG2: symbol dictionary height class at height {height} codes no symbols.");
+ }
+
+ // §6.5.10: with SDHUFF = 1 the export run lengths use table B.1.
+ var runs = HuffmanTable.Get(1);
+ var selector = new ExportSelector(inputSymbols, newSymbols, p.ExportedSymbols);
+ while (!selector.Done) selector.Take(runs.Decode(ref reader));
+ return selector.Result();
+ }
+
+ ///
+ /// §6.5.9: a height class's bitmaps, side by side in one collective bitmap
+ /// TOTWIDTH wide, byte-aligned, BMSIZE bytes long. BMSIZE 0 means stored
+ /// uncompressed, row by row, each row padded to a byte; anything else is that
+ /// many bytes of MMR.
+ ///
+ private static void DecodeCollective(
+ ref Jbig2BitReader reader, HuffmanTables tables, List widths, long totalWidth, int height,
+ List newSymbols, Jbig2PixelBudget budget)
+ {
+ var size = tables.BitmapSize.Decode(ref reader);
+ if (size == HuffmanTable.OutOfBand || size < 0)
+ throw new InvalidDataException("JBIG2: symbol dictionary collective bitmap size is not a non-negative integer.");
+
+ if (totalWidth > int.MaxValue || totalWidth * height > Jbig2Limits.MaxBitmapPixels)
+ throw new InvalidDataException(
+ $"JBIG2: symbol dictionary height class is {totalWidth}x{height}, past the " +
+ $"{Jbig2Limits.MaxBitmapPixels:N0} pixel ceiling for one bitmap.");
+
+ reader.AlignToByte();
+ var start = reader.BytePosition;
+ var data = reader.Data;
+ var w = (int)totalWidth;
+ Jbig2Bitmap collective;
+ long length;
+
+ if (size == 0)
+ {
+ var stride = (w + 7) / 8;
+ length = (long)stride * height;
+ if (start + length > data.Length)
+ throw new InvalidDataException("JBIG2: uncompressed symbol dictionary collective bitmap ends early.");
+
+ budget.Charge(w, height);
+ collective = new Jbig2Bitmap(w, height);
+ for (var y = 0; y < height; y++)
+ {
+ var row = data.Slice(start + y * stride, stride);
+ for (var x = 0; x < w; x++)
+ collective.Data[y * w + x] = (byte)((row[x >> 3] >> (7 - (x & 7))) & 1);
+ }
+ }
+ else
+ {
+ length = size;
+ if (start + length > data.Length)
+ throw new InvalidDataException("JBIG2: MMR symbol dictionary collective bitmap ends early.");
+ collective = MmrDecoder.Decode(data.Slice(start, size), w, height, budget);
+ }
+
+ reader.SeekToByte(start + length);
+
+ var x0 = 0;
+ foreach (var width in widths)
+ {
+ newSymbols.Add(collective.Crop(x0, 0, width, height));
+ x0 += width;
+ }
+ }
+
+ /// §6.5.8.2 with SDHUFF = 1: a refined or aggregated symbol.
+ private static Jbig2Bitmap DecodeAggregateHuffman(
+ ref Jbig2BitReader reader,
+ SymbolDictionaryParameters p,
+ Jbig2Bitmap[] inputSymbols,
+ List newSymbols,
+ int width,
+ int height,
+ int codeLength,
+ HuffmanTables tables,
+ Span refinementContexts,
+ scoped ReadOnlySpan refinementAt,
+ Jbig2PixelBudget budget)
+ {
+ var instances = tables.AggregateInstances.Decode(ref reader);
+ if (instances == HuffmanTable.OutOfBand || instances < 1)
+ throw new InvalidDataException("JBIG2: symbol dictionary aggregate instance count is not a positive integer.");
+
+ var alphabet = new Jbig2Bitmap[inputSymbols.Length + p.NewSymbols];
+ inputSymbols.CopyTo(alphabet, 0);
+ newSymbols.CopyTo(alphabet, inputSymbols.Length);
+ for (var i = inputSymbols.Length + newSymbols.Count; i < alphabet.Length; i++)
+ alphabet[i] = Jbig2Bitmap.Empty;
+
+ if (instances == 1)
+ {
+ // §6.5.8.2.2: ID as SBSYMCODELEN raw bits, RDX and RDY with B.15, the
+ // refinement's byte count with B.1, then the refinement itself,
+ // arithmetically coded, starting on the next byte.
+ var id = (int)reader.ReadBits(codeLength);
+ if ((uint)id >= (uint)alphabet.Length)
+ throw new InvalidDataException($"JBIG2: aggregate symbol names symbol {id} of {alphabet.Length}.");
+
+ var rdx = Require(HuffmanTable.Get(15).Decode(ref reader), "RDX");
+ var rdy = Require(HuffmanTable.Get(15).Decode(ref reader), "RDY");
+ var size = Require(HuffmanTable.Get(1).Decode(ref reader), "BMSIZE");
+
+ return TextRegionDecoder.DecodeHuffmanRefinement(
+ ref reader, size, refinementContexts, width, height, p.RefinementTemplate,
+ alphabet[id], rdx, rdy, refinementAt, budget);
+ }
+
+ // §6.5.8.2.1: a text region inside the dictionary, with the tables Table 17
+ // fixes and uniform-length symbol IDs.
+ var parameters = new TextRegionParameters(
+ Width: width,
+ Height: height,
+ Instances: instances,
+ Strips: 1,
+ DefaultPixel: 0,
+ Combination: CombinationOperator.Or,
+ Corner: ReferenceCorner.TopLeft,
+ Transposed: false,
+ DsOffset: 0,
+ Refine: true,
+ RefinementTemplate: p.RefinementTemplate);
+
+ return TextRegionDecoder.DecodeHuffman(
+ ref reader, parameters, alphabet, TextRegionDecoder.HuffmanTables.SymbolDictionaryAggregate,
+ symbolCodes: null, codeLength, refinementContexts, refinementAt, budget);
}
///
@@ -206,57 +425,49 @@ private static Jbig2Bitmap DecodeAggregate(
///
/// §6.5.10: which symbols leave the dictionary, as alternating runs of
/// "skip this many" and "export this many" over the imports followed by the
- /// new symbols.
+ /// new symbols. Fed one run at a time, so the arithmetic and Huffman paths can
+ /// share it whatever decodes their runs.
///
- private static Jbig2Bitmap[] SelectExported(
- ref MqDecoder mq,
- ArithIntDecoder ex,
- Jbig2Bitmap[] inputSymbols,
- List newSymbols,
- int expected)
+ private sealed class ExportSelector(Jbig2Bitmap[] inputSymbols, List newSymbols, int expected)
{
- var total = inputSymbols.Length + newSymbols.Count;
- var exported = new List(expected);
- var index = 0;
- var exporting = false;
-
- // The same non-termination trap as the height-class loop above: a run of
- // length zero advances nothing, and a never-dry MQ decoder can yield zero
- // for ever. A *leading* zero run is completely normal — a dictionary that
- // exports from index 0 codes one — so this is a ceiling on the number of
- // runs rather than a ban on empty ones. Every run that advances costs at
- // least one symbol, so any real encoding is far below the bound.
- var runs = 0;
- var maxRuns = 2 * total + 8;
-
- while (index < total)
+ private readonly int _total = inputSymbols.Length + newSymbols.Count;
+ private readonly List _exported = new(Math.Min(expected, 1 << 16));
+ private int _index;
+ private bool _exporting;
+
+ // A zero-length run advances nothing, and a never-dry MQ decoder can yield
+ // zero for ever. A *leading* zero run is completely normal (a dictionary
+ // that exports from index 0 codes one), so this is a ceiling on the number
+ // of runs rather than a ban on empty ones.
+ private int _runs;
+
+ public bool Done => _index >= _total;
+
+ public void Take(int run)
{
- if (++runs > maxRuns)
+ if (++_runs > 2 * _total + 8)
throw new InvalidDataException(
- $"JBIG2: symbol dictionary export runs do not terminate — more than {maxRuns} runs " +
- $"over {total} symbols.");
-
- var run = ex.Decode(ref mq);
+ $"JBIG2: symbol dictionary export runs do not terminate — more than {2 * _total + 8} runs " +
+ $"over {_total} symbols.");
if (run == ArithIntDecoder.OutOfBand || run < 0)
throw new InvalidDataException("JBIG2: symbol dictionary export run length is not a non-negative integer.");
- if (index + run > total)
+ if (_index + run > _total)
throw new InvalidDataException("JBIG2: symbol dictionary export runs overrun the symbol list.");
- if (exporting)
+ if (_exporting)
for (var i = 0; i < run; i++)
- exported.Add(index + i < inputSymbols.Length
- ? inputSymbols[index + i]
- : newSymbols[index + i - inputSymbols.Length]);
+ _exported.Add(_index + i < inputSymbols.Length
+ ? inputSymbols[_index + i]
+ : newSymbols[_index + i - inputSymbols.Length]);
- index += run;
- exporting = !exporting;
+ _index += run;
+ _exporting = !_exporting;
}
- if (exported.Count != expected)
- throw new InvalidDataException(
- $"JBIG2: symbol dictionary exported {exported.Count} symbols but declared {expected}.");
-
- return [.. exported];
+ public Jbig2Bitmap[] Result() => _exported.Count == expected
+ ? [.. _exported]
+ : throw new InvalidDataException(
+ $"JBIG2: symbol dictionary exported {_exported.Count} symbols but declared {expected}.");
}
private static int Require(int value, string field) => value != ArithIntDecoder.OutOfBand
diff --git a/src/Sdcb.SimdCodecs.Jbig2/TextRegionDecoder.cs b/src/Sdcb.SimdCodecs.Jbig2/TextRegionDecoder.cs
index 365d27d..6bb07e5 100644
--- a/src/Sdcb.SimdCodecs.Jbig2/TextRegionDecoder.cs
+++ b/src/Sdcb.SimdCodecs.Jbig2/TextRegionDecoder.cs
@@ -52,7 +52,8 @@ internal readonly record struct TextRegionParameters(
/// Every number here comes from an — one per field,
/// as T.88 names them (IADT, IAFS, IADS, IAIT, IARI, IARDW…), because each
/// field's values have their own distribution and sharing contexts between them
-/// would cost compression.
+/// would cost compression. A Huffman-coded region (SBHUFF = 1) reads the same
+/// fields from Huffman tables instead, in .
///
///
internal static class TextRegionDecoder
@@ -228,6 +229,203 @@ private static int PlaceSymbol(Jbig2Bitmap region, Jbig2Bitmap symbol, TextRegio
return advanceFirst ? curs : curs + extent - 1;
}
+ /// The Huffman tables a Huffman-coded text region reads its fields with (§7.4.3.1.2).
+ internal sealed record HuffmanTables(
+ HuffmanTable Fs, HuffmanTable Ds, HuffmanTable Dt,
+ HuffmanTable Rdw, HuffmanTable Rdh, HuffmanTable Rdx, HuffmanTable Rdy, HuffmanTable RefinementSize)
+ {
+ ///
+ /// The tables T.88 Table 17 fixes for a text region decoded inside a
+ /// Huffman-coded symbol dictionary's aggregate symbol (§6.5.8.2.1).
+ ///
+ public static HuffmanTables SymbolDictionaryAggregate => new(
+ HuffmanTable.Get(6), HuffmanTable.Get(8), HuffmanTable.Get(11),
+ HuffmanTable.Get(15), HuffmanTable.Get(15), HuffmanTable.Get(15), HuffmanTable.Get(15),
+ HuffmanTable.Get(1));
+ }
+
+ ///
+ /// Reads the symbol ID Huffman table that opens a Huffman-coded text region's data
+ /// (§7.4.3.1.7): 35 four-bit lengths for the run codes, then one code length per
+ /// symbol coded with those run codes, then padding to the next byte.
+ ///
+ public static HuffmanTable ReadSymbolCodeTable(ref Jbig2BitReader reader, int symbolCount)
+ {
+ Span runLengths = stackalloc int[35];
+ for (var i = 0; i < runLengths.Length; i++) runLengths[i] = (int)reader.ReadBits(4);
+ var runCodes = HuffmanTable.FromCodeLengths(runLengths);
+
+ var lengths = new int[symbolCount];
+ var count = 0;
+ while (count < symbolCount)
+ {
+ var code = runCodes.Decode(ref reader);
+ int repeat, value;
+ switch (code)
+ {
+ case >= 0 and < 32:
+ lengths[count++] = code;
+ continue;
+ case 32:
+ // RUNCODE32: the previous length again, 3 to 6 times.
+ if (count == 0)
+ throw new InvalidDataException("JBIG2: symbol ID table repeats a length before any is given.");
+ value = lengths[count - 1];
+ repeat = 3 + (int)reader.ReadBits(2);
+ break;
+ case 33:
+ value = 0;
+ repeat = 3 + (int)reader.ReadBits(3);
+ break;
+ case 34:
+ value = 0;
+ repeat = 11 + (int)reader.ReadBits(7);
+ break;
+ default:
+ throw new InvalidDataException($"JBIG2: symbol ID table has run code {code}.");
+ }
+
+ if (count + repeat > symbolCount)
+ throw new InvalidDataException("JBIG2: symbol ID table runs past the symbol count.");
+ lengths.AsSpan(count, repeat).Fill(value);
+ count += repeat;
+ }
+
+ reader.AlignToByte();
+ return HuffmanTable.FromCodeLengths(lengths);
+ }
+
+ ///
+ /// Decodes a Huffman-coded text region (SBHUFF = 1, §6.4.5 with the Huffman
+ /// branches of §6.4.6 to §6.4.11).
+ ///
+ /// Positioned at the first field after the symbol ID table.
+ /// The region parameters.
+ /// SBSYMS.
+ /// The field tables.
+ ///
+ /// SBSYMCODES, or null for the uniform -bit IDs a symbol
+ /// dictionary's aggregate region uses.
+ ///
+ /// The width of a uniform ID; unused when is given.
+ /// Contexts for per-instance refinement.
+ /// SBRAT.
+ /// The decode's remaining pixel allowance.
+ public static Jbig2Bitmap DecodeHuffman(
+ ref Jbig2BitReader reader,
+ TextRegionParameters p,
+ Jbig2Bitmap[] symbols,
+ HuffmanTables tables,
+ HuffmanTable? symbolCodes,
+ int codeLength,
+ Span refinementContexts,
+ scoped ReadOnlySpan refinementAt,
+ Jbig2PixelBudget budget)
+ {
+ budget.Charge(p.Width, p.Height);
+ var region = new Jbig2Bitmap(p.Width, p.Height, p.DefaultPixel);
+ var logStrips = System.Numerics.BitOperations.Log2((uint)p.Strips);
+
+ var stript = -(long)RequireHuffman(tables.Dt.Decode(ref reader), "DT") * p.Strips;
+ long firsts = 0;
+ var instances = 0;
+
+ while (instances < p.Instances)
+ {
+ stript += (long)RequireHuffman(tables.Dt.Decode(ref reader), "DT") * p.Strips;
+ firsts += RequireHuffman(tables.Fs.Decode(ref reader), "FS");
+ var curs = firsts;
+
+ while (true)
+ {
+ if (instances >= p.Instances)
+ throw new InvalidDataException("JBIG2: text region codes more symbol instances than it declared.");
+
+ // §6.4.9: with SBHUFF the within-strip T offset is LOG2SBSTRIPS raw bits.
+ var curt = (long)reader.ReadBits(logStrips);
+ var t = ClampCoordinate(stript + curt);
+
+ var id = symbolCodes is null ? (int)reader.ReadBits(codeLength) : symbolCodes.Decode(ref reader);
+ if ((uint)id >= (uint)symbols.Length)
+ throw new InvalidDataException($"JBIG2: text region names symbol {id} of {symbols.Length}.");
+
+ var symbol = symbols[id];
+ if (p.Refine && reader.ReadBit() != 0)
+ {
+ var rdw = RequireHuffman(tables.Rdw.Decode(ref reader), "RDW");
+ var rdh = RequireHuffman(tables.Rdh.Decode(ref reader), "RDH");
+ var rdx = RequireHuffman(tables.Rdx.Decode(ref reader), "RDX");
+ var rdy = RequireHuffman(tables.Rdy.Decode(ref reader), "RDY");
+ var size = RequireHuffman(tables.RefinementSize.Decode(ref reader), "RSIZE");
+
+ var width = (long)symbol.Width + rdw;
+ var height = (long)symbol.Height + rdh;
+ if (width is <= 0 or > int.MaxValue || height is <= 0 or > int.MaxValue)
+ throw new InvalidDataException($"JBIG2: refined text instance has size {width}x{height}.");
+
+ symbol = DecodeHuffmanRefinement(
+ ref reader, size, refinementContexts, (int)width, (int)height, p.RefinementTemplate,
+ symbol, FloorHalf(rdw) + rdx, FloorHalf(rdh) + rdy, refinementAt, budget);
+ }
+
+ curs = PlaceSymbol(region, symbol, p, ClampCoordinate(curs), t);
+ instances++;
+
+ var ds = tables.Ds.Decode(ref reader);
+ if (ds == HuffmanTable.OutOfBand) break;
+
+ curs += (long)ds + p.DsOffset;
+ }
+ }
+
+ return region;
+ }
+
+ ///
+ /// A refinement inside Huffman-coded data (§6.4.11.1 and §6.5.8.2.2): it starts
+ /// on the next byte boundary, is arithmetically coded, and occupies exactly
+ /// bytes, after which the Huffman fields resume.
+ ///
+ public static Jbig2Bitmap DecodeHuffmanRefinement(
+ ref Jbig2BitReader reader,
+ int size,
+ Span contexts,
+ int width,
+ int height,
+ int template,
+ Jbig2Bitmap reference,
+ int dx,
+ int dy,
+ scoped ReadOnlySpan at,
+ Jbig2PixelBudget budget)
+ {
+ if (size < 0) throw new InvalidDataException($"JBIG2: refinement data size {size} is negative.");
+
+ reader.AlignToByte();
+ var start = reader.BytePosition;
+ if ((long)start + size > reader.Data.Length)
+ throw new InvalidDataException("JBIG2: Huffman-coded refinement data runs past the segment.");
+
+ // Each refinement is its own arithmetically coded unit, so it starts from
+ // fresh adaptive state.
+ contexts.Clear();
+ var mq = new MqDecoder(reader.Data.Slice(start, size));
+ var bitmap = RefinementRegionDecoder.Decode(
+ ref mq, contexts, width, height, template, reference, dx, dy,
+ typicalPrediction: false, at, budget);
+
+ reader.SeekToByte((long)start + size);
+ return bitmap;
+ }
+
+ // Coordinates are accumulated in long from coded deltas; anything outside int
+ // can only place a symbol off the region, so pin it somewhere that still is.
+ private static int ClampCoordinate(long value) => (int)Math.Clamp(value, int.MinValue / 2, int.MaxValue / 2);
+
+ private static int RequireHuffman(int value, string field) => value != HuffmanTable.OutOfBand
+ ? value
+ : throw new InvalidDataException($"JBIG2: unexpected OOB decoding {field} in a text region.");
+
/// Floor division by two that keeps working for negative deltas, which / 2 does not.
private static int FloorHalf(int value) => value >> 1;
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/BlockDecoder.cs b/src/Sdcb.SimdCodecs.Jpeg2000/BlockDecoder.cs
index 07ec114..13f8380 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/BlockDecoder.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/BlockDecoder.cs
@@ -81,35 +81,33 @@ private static void DecodePasses(
var mq = new MqDecoder(coded);
Span contexts = stackalloc byte[ContextCount];
- contexts.Clear();
-
- // The three states T.800 Table D.7 seeds away from zero. This is the
- // roadmap's hazard 1: the coder is shared with JBIG2 but its
- // initialisation is not, and getting it wrong decodes the first
- // code-block plausibly and then drifts.
- contexts[0] = 4 << 1; // zero coding, all-insignificant neighbourhood
- contexts[RunLengthContext] = 3 << 1;
- contexts[UniformContext] = 46 << 1;
+ ResetContexts(contexts);
var passesLeft = block.PassCount;
var plane = planes - 1;
+ var reset = (band.CodeBlockStyle & CodeBlockStyles.ResetContexts) != 0;
+ var segmentation = (band.CodeBlockStyle & CodeBlockStyles.SegmentationSymbols) != 0;
// The first pass of a code-block is always a cleanup pass: there is
// nothing significant yet for a significance-propagation pass to
// propagate from, and nothing to refine.
- state.CleanupPass(ref mq, contexts, plane);
+ Cleanup(ref mq, contexts, state, plane, segmentation);
+ if (reset) ResetContexts(contexts);
passesLeft--;
plane--;
while (plane >= 0 && passesLeft > 0)
{
state.SignificancePropagationPass(ref mq, contexts, plane);
+ if (reset) ResetContexts(contexts);
if (--passesLeft == 0) break;
state.MagnitudeRefinementPass(ref mq, contexts, plane);
+ if (reset) ResetContexts(contexts);
if (--passesLeft == 0) break;
- state.CleanupPass(ref mq, contexts, plane);
+ Cleanup(ref mq, contexts, state, plane, segmentation);
+ if (reset) ResetContexts(contexts);
passesLeft--;
plane--;
}
@@ -117,6 +115,37 @@ private static void DecodePasses(
state.WriteTo(band, block);
}
+ ///
+ /// The cleanup pass, followed when the code-block style asks for it by the segmentation symbol
+ /// of D.5: four decisions in the uniform context, 1010 in a sound codestream. They carry no
+ /// coefficient data, only a check an error-resilient decoder could make, so they are decoded to
+ /// keep the codeword in step and their value is not acted on.
+ ///
+ private static void Cleanup(ref MqDecoder mq, scoped Span contexts, BlockState state, int plane, bool segmentation)
+ {
+ state.CleanupPass(ref mq, contexts, plane);
+ if (!segmentation) return;
+
+ for (var i = 0; i < 4; i++) mq.Decode(contexts, UniformContext);
+ }
+
+ ///
+ /// The initial states of T.800 Table D.7: every context at state 0 bar three. Called once
+ /// before the first pass, and after every pass when the code-block style resets contexts (D.4.1).
+ ///
+ private static void ResetContexts(Span contexts)
+ {
+ contexts.Clear();
+
+ // The three states T.800 Table D.7 seeds away from zero. This is the
+ // roadmap's hazard 1: the coder is shared with JBIG2 but its
+ // initialisation is not, and getting it wrong decodes the first
+ // code-block plausibly and then drifts.
+ contexts[0] = 4 << 1; // zero coding, all-insignificant neighbourhood
+ contexts[RunLengthContext] = 3 << 1;
+ contexts[UniformContext] = 46 << 1;
+ }
+
///
/// Concatenates the block's coded segments.
///
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/CodestreamReader.cs b/src/Sdcb.SimdCodecs.Jpeg2000/CodestreamReader.cs
index 8aa2421..3203808 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/CodestreamReader.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/CodestreamReader.cs
@@ -51,7 +51,7 @@ internal sealed record CodestreamHeader(
///
/// The distinction kept throughout: means the
/// bytes are malformed or hostile, means
-/// they are a perfectly legal codestream using a feature this rung does not
+/// they are a perfectly legal codestream using a feature this decoder does not
/// implement yet. A caller can reasonably retry the second with another decoder
/// and cannot do anything with the first.
///
@@ -232,7 +232,7 @@ private static void ReadTilePart(ReadOnlySpan data, ref int offset, List data, ref int offset, List body)
///
/// Refuses, by name, everything outside the envelope this decoder has actually
- /// been validated against. Each message says which rung owns the feature, so
- /// the refusal reads as a roadmap position rather than a dead end.
+ /// been validated against, naming the feature in the message.
///
private static void Validate(CodestreamHeader header)
{
@@ -522,23 +521,18 @@ private static void Validate(CodestreamHeader header)
$"{first.BitDepth}-bit. Components of different precisions are not implemented.");
}
- if (siz.TileCount != 1)
- throw new NotSupportedException(
- $"JPEG 2000: this codestream has {siz.TileCount} tiles; only single-tile images are " +
- "implemented. Multiple tiles are rung 3.");
- if (header.TileParts.Count != 1)
- throw new NotSupportedException(
- $"JPEG 2000: this tile is split into {header.TileParts.Count} tile-parts, which is rung 3.");
-
- if (header.Progression != ProgressionOrder.Lrcp)
- throw new NotSupportedException(
- $"JPEG 2000: progression order {header.Progression} is not implemented; only LRCP is. " +
- "With one tile and maximal precincts the five orders differ only in how they nest layer, " +
- "resolution and component, and only LRCP's nesting has a fixture behind it.");
+ // A.4.2: Isot is 16 bits, so a codestream cannot address more tiles than that; one that
+ // declares more is not a JPEG 2000 codestream.
+ if (siz.TileCount > 65535)
+ throw new InvalidDataException(
+ $"JPEG 2000: SIZ declares {siz.TileCount:N0} tiles, more than the 65,535 a tile index can name.");
- if (header.UseSopMarkers || header.UseEphMarkers)
- throw new NotSupportedException(
- "JPEG 2000: SOP and EPH markers are not implemented; they are rung 3.");
+ foreach (var part in header.TileParts)
+ {
+ if (part.TileIndex >= siz.TileCount)
+ throw new InvalidDataException(
+ $"JPEG 2000: a tile-part names tile {part.TileIndex}, but SIZ declares only {siz.TileCount}.");
+ }
for (var c = 0; c < siz.Components.Length; c++)
{
@@ -559,18 +553,28 @@ private static void Validate(CodestreamHeader header)
"sizes. Only scalar quantization, derived or expounded, is implemented for the 9/7 " +
"wavelet.");
- if (coding.CodeBlockStyle != 0)
+ // T.800 Table A.19. Context reset (0x02), predictable termination (0x10, which only
+ // constrains how the encoder terminates) and segmentation symbols (0x20) are decoded.
+ // Selective arithmetic bypass, termination on every pass and the vertically causal
+ // context each change how the coded bytes are read, so ignoring one would desynchronise
+ // tier-1 rather than degrade quality.
+ var unsupported = coding.CodeBlockStyle & ~(CodeBlockStyles.ResetContexts |
+ CodeBlockStyles.PredictableTermination | CodeBlockStyles.SegmentationSymbols);
+ if (unsupported != 0)
throw new NotSupportedException(
- $"JPEG 2000: code-block style flags 0x{coding.CodeBlockStyle:X2} are not implemented " +
- "(T.800 Table A.19: selective arithmetic bypass, context reset, termination per pass, " +
- "vertically causal context, predictable termination, segmentation symbols). Each changes " +
- "how the coded bytes are read, so ignoring one desynchronises tier-1 rather than degrading " +
- "quality.");
+ $"JPEG 2000: code-block style flags 0x{unsupported:X2} are not implemented (T.800 Table " +
+ "A.19: 0x01 selective arithmetic bypass, 0x04 termination on each pass, 0x08 vertically " +
+ "causal context). Each changes how the coded bytes are read.");
- if (coding.PrecinctSizes.Length != 0)
- throw new NotSupportedException(
- "JPEG 2000: custom precinct sizes are not implemented; only the maximal default, which " +
- "makes each resolution a single precinct. Precincts are rung 3.");
+ // A.6.1: a precinct exponent of 0 is allowed only at resolution 0. Above it the bands see
+ // the precinct at half size, which a zero exponent cannot be halved into.
+ for (var r = 1; r < coding.ResolutionCount; r++)
+ {
+ if (coding.PrecinctWidthExponent(r) == 0 || coding.PrecinctHeightExponent(r) == 0)
+ throw new InvalidDataException(
+ $"JPEG 2000: component {c} declares a precinct exponent of 0 at resolution {r}, " +
+ "which T.800 allows only at resolution 0.");
+ }
}
if (header.MultipleComponentTransform)
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/Geometry.cs b/src/Sdcb.SimdCodecs.Jpeg2000/Geometry.cs
index 237417d..be61f23 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/Geometry.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/Geometry.cs
@@ -56,8 +56,8 @@ internal readonly record struct Rect(int X0, int Y0, int X1, int Y1)
///
/// Mutable, and deliberately so — ,
/// and persist across the packets of successive
-/// quality layers, which is the state that makes tier-2 stateful. Rung 1 has one
-/// layer and so cannot exercise it, which is exactly why hazard 5 warns that a
+/// quality layers, which is the state that makes tier-2 stateful. A one-layer
+/// codestream cannot exercise it, which is exactly why hazard 5 warns that a
/// single-layer fixture proves nothing about this.
///
///
@@ -142,35 +142,20 @@ internal sealed class Subband
///
public required float[] Values { get; init; }
- ///
- /// The precinct's inclusion tag tree: for each code-block, the quality layer
- /// that first carries it.
- ///
- public TagTree? InclusionTree { get; private set; }
+ /// T.800 Table A.19 code-block style flags of the tile-component this band belongs to.
+ public required int CodeBlockStyle { get; init; }
- ///
- /// The precinct's imsb tag tree: for each code-block, how many leading
- /// magnitude bit-planes are entirely zero.
- ///
- public TagTree? ZeroBitPlaneTree { get; private set; }
+ /// xcb', the code-block width exponent after the precinct cap of Equation B-16.
+ public required int BlockWidthExponent { get; init; }
- ///
- /// Creates the pair once, on first use.
- ///
- /// One pair per band is correct only because rung 1 has maximal
- /// precincts, which makes each resolution a single precinct. The trees
- /// genuinely belong to a precinct, and rung 3 must move them there when
- /// custom precinct sizes arrive. What must not change either way is the
- /// lifetime: they persist across the packets of successive quality layers,
- /// and rebuilding them per packet decodes layer 0 correctly and everything
- /// after it wrongly.
- ///
- ///
- public void EnsureTagTrees()
- {
- InclusionTree ??= new TagTree(BlocksWide, BlocksHigh);
- ZeroBitPlaneTree ??= new TagTree(BlocksWide, BlocksHigh);
- }
+ /// ycb', the code-block height exponent after the precinct cap.
+ public required int BlockHeightExponent { get; init; }
+
+ /// The absolute column of the first code-block, on the grid anchored at the band origin.
+ public required int FirstBlockColumn { get; init; }
+
+ /// The absolute row of the first code-block.
+ public required int FirstBlockRow { get; init; }
}
///
@@ -187,6 +172,71 @@ internal sealed class Resolution
/// One band (LL) at level 0, three (HL, LH, HH) above it.
public required Subband[] Bands { get; init; }
+
+ ///
+ /// The resolution's precincts in raster order (T.800 B.6), which is the order their index k
+ /// counts them in. Empty when the resolution itself is empty: such a resolution has no packets.
+ ///
+ public required Precinct[] Precincts { get; init; }
+}
+
+///
+/// One precinct of one resolution (T.800 B.6): the unit a packet carries. It holds, for each band
+/// of its resolution, the rectangle of code-blocks that fall inside it and that band's two tag
+/// trees, which belong to the precinct and persist across the packets of successive layers.
+///
+internal sealed class Precinct
+{
+ /// One entry per band of the resolution, in the band order the packet codes them.
+ public required PrecinctBand[] Bands { get; init; }
+
+ ///
+ /// Where the position-driven progressions (RPCL, PCRL, CPRL; T.800 B.12.1.3 to B.12.1.5) visit
+ /// this precinct on the reference grid: the precinct's top-left corner scaled up to the reference
+ /// grid, clipped to the tile's corner. Those orders walk the reference grid and take a precinct
+ /// at the first point that falls on it, which is exactly this point.
+ ///
+ public required long VisitX { get; init; }
+
+ /// The vertical counterpart of .
+ public required long VisitY { get; init; }
+}
+
+/// The code-blocks one precinct takes from one band, and that band's tag trees for the precinct.
+internal sealed class PrecinctBand
+{
+ /// The band.
+ public required Subband Band { get; init; }
+
+ /// The first code-block column, as an index into ' grid.
+ public required int BlockX0 { get; init; }
+
+ /// The first code-block row, likewise.
+ public required int BlockY0 { get; init; }
+
+ /// Code-blocks across this precinct in this band; zero when none fall inside it.
+ public required int BlocksWide { get; init; }
+
+ /// Code-blocks down.
+ public required int BlocksHigh { get; init; }
+
+ ///
+ /// The inclusion tag tree over this precinct's code-blocks: for each, the layer that first carries it.
+ ///
+ public TagTree? InclusionTree { get; private set; }
+
+ /// The imsb tag tree: for each code-block, how many leading magnitude bit-planes are zero.
+ public TagTree? ZeroBitPlaneTree { get; private set; }
+
+ ///
+ /// Creates the pair once, on first use. They persist across the packets of successive quality
+ /// layers: rebuilding them per packet decodes layer 0 correctly and every later layer wrongly.
+ ///
+ public void EnsureTagTrees()
+ {
+ InclusionTree ??= new TagTree(BlocksWide, BlocksHigh);
+ ZeroBitPlaneTree ??= new TagTree(BlocksWide, BlocksHigh);
+ }
}
///
@@ -194,7 +244,7 @@ internal sealed class Resolution
/// tier-2 and the DWT both need (T.800 Annex B).
///
/// Everything here is computed from declared numbers before a coded byte is
-/// read, which is why is where the resource ceilings are
+/// read, which is why is where the resource ceilings are
/// charged. Hazard 6's point exactly: a SIZ and a COD alone can
/// ask for an arbitrarily large object graph, and no amount of running out of
/// input will stop it.
@@ -220,30 +270,41 @@ internal sealed class TileComponent
public required int DecodedResolutions { get; init; }
///
- /// Builds the geometry for one component of a single-tile codestream, from that component's own
- /// coding style and quantization (COD and QCD with its COC and QCC applied).
+ /// for the tile that
+ /// covers the whole image region: the only tile of a single-tile codestream.
+ ///
+ public static TileComponent Build(
+ CodestreamHeader header, int componentIndex, Jpeg2000SampleBudget budget, int decodedResolutions)
+ {
+ var siz = header.Siz;
+ return Build(header, componentIndex, new Rect(siz.X0, siz.Y0, siz.X1, siz.Y1), budget, decodedResolutions);
+ }
+
+ ///
+ /// Builds the geometry for one component of one tile, from that component's own coding style
+ /// and quantization (COD and QCD with its COC and QCC applied).
///
+ /// The tile's extent on the reference grid (T.800 Equation B-7).
///
/// How many resolution levels from 0 will be reconstructed; only their subbands get coefficient
/// storage, and only theirs are charged to the budget as samples.
///
public static TileComponent Build(
- CodestreamHeader header, int componentIndex, Jpeg2000SampleBudget budget, int decodedResolutions)
+ CodestreamHeader header, int componentIndex, Rect tile, Jpeg2000SampleBudget budget, int decodedResolutions)
{
var siz = header.Siz;
var cod = header.ComponentCoding[componentIndex];
var quantization = header.ComponentQuantization[componentIndex];
var component = siz.Components[componentIndex];
- // T.800 Equation B-12. With one tile the tile IS the image region, and
- // with no subsampling the separation divides out to 1 -- but the formula
- // is written in full because the coordinates, not the sizes, are what
- // later equations consume.
+ // T.800 Equation B-12: the tile divided by the component's subsampling, rounding up. The
+ // formula is written in full because the coordinates, not the sizes, are what later
+ // equations consume.
var bounds = new Rect(
- CeilDiv(siz.X0, component.HorizontalSeparation),
- CeilDiv(siz.Y0, component.VerticalSeparation),
- CeilDiv(siz.X1, component.HorizontalSeparation),
- CeilDiv(siz.Y1, component.VerticalSeparation));
+ CeilDiv(tile.X0, component.HorizontalSeparation),
+ CeilDiv(tile.Y0, component.VerticalSeparation),
+ CeilDiv(tile.X1, component.HorizontalSeparation),
+ CeilDiv(tile.Y1, component.VerticalSeparation));
if (bounds.Area > Jpeg2000Limits.MaxTileComponentSamples)
throw new InvalidDataException(
@@ -301,7 +362,13 @@ public static TileComponent Build(
bands[b] = band;
}
- resolutions[r] = new Resolution { Index = r, Bounds = resolutionBounds, Bands = bands };
+ resolutions[r] = new Resolution
+ {
+ Index = r,
+ Bounds = resolutionBounds,
+ Bands = bands,
+ Precincts = BuildPrecincts(resolutionBounds, bands, cod, r, levels, tile, component, budget),
+ };
}
return new TileComponent
@@ -313,6 +380,88 @@ public static TileComponent Build(
};
}
+ ///
+ /// T.800 B.6: partitions a resolution into precincts of 2^PPx x 2^PPy on the resolution grid,
+ /// anchored at its origin, and finds, for each precinct, the code-blocks of every band that fall
+ /// inside it. A band above resolution 0 sees the precinct at half size, 2^(PPx-1), because band
+ /// coordinates are resolution coordinates halved; and since Equation B-16 caps the code-block size
+ /// at that, code-blocks nest inside precincts exactly.
+ ///
+ private static Precinct[] BuildPrecincts(
+ Rect resolution, Subband[] bands, CodingStyle cod, int r, int levels, Rect tile, ComponentInfo component,
+ Jpeg2000SampleBudget budget)
+ {
+ if (resolution.IsEmpty) return [];
+
+ var ppx = cod.PrecinctWidthExponent(r);
+ var ppy = cod.PrecinctHeightExponent(r);
+ var firstX = FloorDiv(resolution.X0, 1 << ppx);
+ var firstY = FloorDiv(resolution.Y0, 1 << ppy);
+ var wide = CeilDiv(resolution.X1, 1 << ppx) - firstX;
+ var high = CeilDiv(resolution.Y1, 1 << ppy) - firstY;
+
+ // A precinct is a handful of objects, so the count is booked against the code-block
+ // ceiling: a header with tiny precincts on a large image must not buy an object graph its
+ // data could never fill.
+ var count = (long)wide * high;
+ if (count > Jpeg2000Limits.MaxCodeBlocks)
+ throw new InvalidDataException(
+ $"JPEG 2000: a resolution declares {count:N0} precincts, more than the " +
+ $"{Jpeg2000Limits.MaxCodeBlocks:N0} ceiling.");
+ budget.ChargeCodeBlocks((int)count);
+
+ var bandPpx = r == 0 ? ppx : ppx - 1;
+ var bandPpy = r == 0 ? ppy : ppy - 1;
+ var scale = levels - r;
+
+ var precincts = new Precinct[count];
+ for (var py = 0; py < high; py++)
+ {
+ for (var px = 0; px < wide; px++)
+ {
+ long column = firstX + px;
+ long row = firstY + py;
+
+ var precinctBands = new PrecinctBand[bands.Length];
+ for (var b = 0; b < bands.Length; b++)
+ {
+ var band = bands[b];
+
+ // The precinct's code-blocks on the band's absolute block grid, clipped to the
+ // blocks the band has.
+ var x0 = Math.Max(band.FirstBlockColumn, (column << bandPpx) >> band.BlockWidthExponent);
+ var x1 = Math.Min(band.FirstBlockColumn + band.BlocksWide,
+ ((column + 1) << bandPpx) >> band.BlockWidthExponent);
+ var y0 = Math.Max(band.FirstBlockRow, (row << bandPpy) >> band.BlockHeightExponent);
+ var y1 = Math.Min(band.FirstBlockRow + band.BlocksHigh,
+ ((row + 1) << bandPpy) >> band.BlockHeightExponent);
+
+ var empty = x1 <= x0 || y1 <= y0;
+ precinctBands[b] = new PrecinctBand
+ {
+ Band = band,
+ BlockX0 = empty ? 0 : (int)(x0 - band.FirstBlockColumn),
+ BlockY0 = empty ? 0 : (int)(y0 - band.FirstBlockRow),
+ BlocksWide = empty ? 0 : (int)(x1 - x0),
+ BlocksHigh = empty ? 0 : (int)(y1 - y0),
+ };
+ }
+
+ // Where B.12.1.3's walk over the reference grid first lands in this precinct: its
+ // corner on the resolution grid times 2^(NL - r) and the subsampling, or the tile's
+ // corner for the precinct that starts before the tile does.
+ precincts[py * wide + px] = new Precinct
+ {
+ Bands = precinctBands,
+ VisitX = Math.Max(tile.X0, ((column << ppx) << scale) * component.HorizontalSeparation),
+ VisitY = Math.Max(tile.Y0, ((row << ppy) << scale) * component.VerticalSeparation),
+ };
+ }
+ }
+
+ return precincts;
+ }
+
private static Subband BuildBand(
BandKind kind, Rect bounds, int exponent, int guardBits, CodingStyle cod, int resolution,
bool irreversible, float stepSize, bool decoded)
@@ -375,6 +524,11 @@ private static Subband BuildBand(
Blocks = blocks,
Irreversible = irreversible,
StepSize = stepSize,
+ CodeBlockStyle = cod.CodeBlockStyle,
+ BlockWidthExponent = blockWidthExponent,
+ BlockHeightExponent = blockHeightExponent,
+ FirstBlockColumn = firstColumn,
+ FirstBlockRow = firstRow,
Coefficients = decoded && !irreversible ? new int[bounds.Area] : [],
Values = decoded && irreversible ? new float[bounds.Area] : [],
};
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/InverseWavelet.cs b/src/Sdcb.SimdCodecs.Jpeg2000/InverseWavelet.cs
index 21c2a19..1d85d1e 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/InverseWavelet.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/InverseWavelet.cs
@@ -248,7 +248,7 @@ private static void Filter(Span signal, int i0, int i1, int[] extended, int
if (length == 1)
{
- RefuseALoneHighPassSample(i0);
+ if ((i0 & 1) != 0) signal[0] >>= 1;
return;
}
@@ -302,7 +302,7 @@ private static void FilterColumns(Span samples, int width, int y0, int y1)
var height = y1 - y0;
if (height <= 1)
{
- if (height == 1) RefuseALoneHighPassSample(y0);
+ if (height == 1 && (y0 & 1) != 0) HalveRow(samples[..width]);
return;
}
@@ -374,7 +374,7 @@ private static void Filter97(Span signal, int i0, int i1, float[] scratch
if (length == 1)
{
- RefuseALoneHighPassSample(i0);
+ if ((i0 & 1) != 0) signal[0] *= 0.5f;
return;
}
@@ -419,7 +419,7 @@ private static void FilterColumns97(Span samples, int width, int y0, int
var height = y1 - y0;
if (height <= 1)
{
- if (height == 1) RefuseALoneHighPassSample(y0);
+ if (height == 1 && (y0 & 1) != 0) HalveRow(samples[..width]);
return;
}
@@ -464,23 +464,22 @@ private static void LiftRows97(Span samples, int width, int height, int f
}
}
- ///
- /// F.3.7's degenerate case. An even index is a lone low-pass sample and passes
- /// through untouched; that happens for any image one sample wide, so it is
- /// ordinary and reachable. An odd index is a lone HIGH-pass sample. The spec
- /// halves it, but reaching this needs an odd image origin, which nothing
- /// available here can encode — so there is no way to check it against the
- /// reference. Refusing beats emitting a number no oracle has ever confirmed;
- /// the whole family's discipline.
- ///
- private static void RefuseALoneHighPassSample(int i0)
+ // F.3.7's degenerate case, a signal one sample long. At an even index it is a lone low-pass
+ // sample and passes through untouched, which happens for any image one sample wide. At an odd
+ // index it is a lone HIGH-pass sample, which the forward transform doubled, so it is halved.
+ // That needs an odd tile or image origin on a resolution one sample across, which a tiled
+ // image whose tile size is odd reaches at its low resolutions.
+
+ /// Halves a one-row signal that is a lone high-pass row, for the reversible filter.
+ private static void HalveRow(Span row)
{
- if ((i0 & 1) == 0) return;
+ for (var x = 0; x < row.Length; x++) row[x] >>= 1;
+ }
- throw new NotSupportedException(
- "JPEG 2000: a subband one sample wide starting at an odd coordinate is not implemented. " +
- "It needs an odd image or tile origin, which opj_compress cannot emit, so this path has " +
- "no reference to be validated against.");
+ /// Halves a one-row signal that is a lone high-pass row, for the irreversible filter.
+ private static void HalveRow(Span row)
+ {
+ for (var x = 0; x < row.Length; x++) row[x] *= 0.5f;
}
///
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/Jp2Reader.cs b/src/Sdcb.SimdCodecs.Jpeg2000/Jp2Reader.cs
index 7a40fbc..f11cdae 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/Jp2Reader.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/Jp2Reader.cs
@@ -1,5 +1,6 @@
using System;
using System.Buffers.Binary;
+using System.Collections.Generic;
using System.IO;
namespace Sdcb.SimdCodecs.Jpeg2000;
@@ -37,19 +38,66 @@ public enum Jp2EnumeratedColourSpace
public sealed record Jp2Colour(int Method, Jp2EnumeratedColourSpace? EnumeratedColourSpace, byte[]? IccProfile);
/// Where a JP2 file's codestream is, and what its header said about colour.
-internal readonly record struct Jp2Layout(int CodestreamStart, int CodestreamLength, Jp2Colour? Colour);
+internal readonly record struct Jp2Layout(int CodestreamStart, int CodestreamLength, Jp2Colour? Colour)
+{
+ ///
+ /// The palette and component mapping boxes, or null when the header has neither and the
+ /// codestream's components are the image's channels as they stand.
+ ///
+ public Jp2ChannelMapping? Channels { get; init; }
+}
+
+///
+/// A JP2 palette (pclr, T.800 I.5.3.4) together with the component mapping (cmap,
+/// I.5.3.5) that says which channel of the image comes from which component, directly or through
+/// which palette column.
+///
+/// NE, the number of palette entries; 0 when there is no palette.
+/// NPC, the number of palette columns.
+/// The bit depth of each palette column.
+/// The palette, Values[entry * Columns + column].
+/// Each output channel: the codestream component, and the palette column or -1 for direct use.
+internal sealed record Jp2ChannelMapping(
+ int Entries, int Columns, int[] ColumnDepths, ushort[] Values, (int Component, int Column)[] Map)
+{
+ ///
+ /// The bit depth of every output channel, which must agree, since the decoded image carries one
+ /// precision. supplies it for the directly mapped channels.
+ ///
+ /// The mapping names a component that does not exist.
+ /// The channels come out at different precisions.
+ public int OutputDepth(IReadOnlyList componentDepths)
+ {
+ var depth = 0;
+ foreach (var (component, column) in Map)
+ {
+ if (component >= componentDepths.Count)
+ throw new InvalidDataException(
+ $"JPEG 2000: the component mapping box names component {component}, but the codestream has " +
+ $"{componentDepths.Count}.");
+
+ var channelDepth = column < 0 ? componentDepths[component] : ColumnDepths[column];
+ if (depth != 0 && channelDepth != depth)
+ throw new NotSupportedException(
+ "JPEG 2000: the JP2 channels come out at different bit depths, and the decoded image has one " +
+ "precision for all of them.");
+ depth = channelDepth;
+ }
+
+ return depth;
+ }
+}
///
/// Reads the JP2 file format of T.800 Annex I far enough to find the codestream and its colour
/// specification: the signature and file type boxes, the header superbox with its image header and
/// colour boxes, and the contiguous codestream box.
///
-/// What is refused, and why each is a refusal rather than something skipped. The palette
-/// (pclr), component mapping (cmap) and channel definition (cdef) boxes each
-/// change what the decoded components mean: a palette turns one component of indices into three of
-/// colour, a mapping reorders them, a channel definition says one is alpha. A reader that skipped
-/// them would hand back samples whose meaning it had silently got wrong, which is the failure this
-/// decoder exists to avoid. None of the JP2 images found in real PDFs carries one.
+/// The palette (pclr) and component mapping (cmap) boxes are read here and applied by
+/// the decoder: a palette turns one component of indices into channels of colour. The channel
+/// definition box (cdef) is refused, because it can say a component is alpha or reorder the
+/// colour channels, and a reader that skipped it would hand back samples whose meaning it had
+/// silently got wrong.
///
///
/// The image header's own width, height and component count are not checked against the
@@ -88,6 +136,7 @@ public static Jp2Layout Read(ReadOnlySpan data)
var sawFileType = false;
var sawHeader = false;
Jp2Colour? colour = null;
+ Jp2ChannelMapping? channels = null;
while (offset < data.Length)
{
@@ -109,7 +158,7 @@ public static Jp2Layout Read(ReadOnlySpan data)
case Header:
if (!sawFileType)
throw new InvalidDataException("JPEG 2000: the JP2 header box comes before the file type box.");
- colour = ReadHeader(body);
+ (colour, channels) = ReadHeader(body);
sawHeader = true;
break;
@@ -118,7 +167,7 @@ public static Jp2Layout Read(ReadOnlySpan data)
throw new InvalidDataException("JPEG 2000: the codestream box comes before the JP2 header box.");
// The first codestream is the image (I.5.4); anything after it is not read.
- return new Jp2Layout(bodyStart, bodyLength, colour);
+ return new Jp2Layout(bodyStart, bodyLength, colour) { Channels = channels };
}
// Every other top-level box (resolution, XML, UUID, IPR) is metadata about the image,
@@ -133,9 +182,11 @@ public static Jp2Layout Read(ReadOnlySpan data)
/// The header superbox: the image header first (I.5.3.1), then at least one colour
/// specification, of which only the first counts (I.5.3.3).
///
- private static Jp2Colour? ReadHeader(ReadOnlySpan header)
+ private static (Jp2Colour? Colour, Jp2ChannelMapping? Channels) ReadHeader(ReadOnlySpan header)
{
Jp2Colour? colour = null;
+ Jp2ChannelMapping? palette = null;
+ (int Component, int Column)[]? map = null;
var offset = 0;
var first = true;
@@ -166,14 +217,12 @@ public static Jp2Layout Read(ReadOnlySpan data)
break;
case Palette:
- throw new NotSupportedException(
- "JPEG 2000: the JP2 file has a palette box (pclr), which turns a component of " +
- "indices into colours. It is not implemented, and ignoring it would return the " +
- "indices as if they were the picture.");
+ palette ??= ReadPalette(body);
+ break;
case ComponentMapping:
- throw new NotSupportedException(
- "JPEG 2000: the JP2 file has a component mapping box (cmap), which is not implemented.");
+ map ??= ReadComponentMapping(body);
+ break;
case ChannelDefinition:
throw new NotSupportedException(
@@ -185,7 +234,115 @@ public static Jp2Layout Read(ReadOnlySpan data)
}
if (first) throw new InvalidDataException("JPEG 2000: the JP2 header box is empty.");
- return colour;
+ return (colour, Combine(palette, map));
+ }
+
+ ///
+ /// I.5.3.4 and I.5.3.5 together. A palette is meaningless without the mapping that says which
+ /// component indexes it, so one without the other is malformed; a mapping without a palette can
+ /// only select components directly.
+ ///
+ private static Jp2ChannelMapping? Combine(Jp2ChannelMapping? palette, (int Component, int Column)[]? map)
+ {
+ if (palette is null && map is null) return null;
+ if (map is null)
+ throw new InvalidDataException("JPEG 2000: the JP2 file has a palette box but no component mapping box.");
+
+ palette ??= new Jp2ChannelMapping(0, 0, [], [], []);
+ foreach (var (_, column) in map)
+ {
+ if (column >= palette.Columns)
+ throw new InvalidDataException(
+ $"JPEG 2000: the component mapping box names palette column {column}, but the palette has " +
+ $"{palette.Columns}.");
+ }
+
+ return palette with { Map = map };
+ }
+
+ ///
+ /// The palette box (I.5.3.4): NE(2) entries and NPC(1) columns, a depth byte per column (the low
+ /// seven bits are the depth less one, the top bit the sign), then the entries one after another,
+ /// each value in as many whole bytes as its column's depth needs.
+ ///
+ private static Jp2ChannelMapping ReadPalette(ReadOnlySpan body)
+ {
+ if (body.Length < 3)
+ throw new InvalidDataException("JPEG 2000: the palette box is too short.");
+
+ var entries = BinaryPrimitives.ReadUInt16BigEndian(body);
+ int columns = body[2];
+
+ // I.5.3.4 bounds NE at 1024 and NPC at 255, and neither below 1. The box is checked against
+ // its own length before anything is sized from it, so the table it allocates is never more
+ // than the bytes that describe it.
+ if (entries is 0 or > 1024 || columns == 0)
+ throw new InvalidDataException(
+ $"JPEG 2000: the palette box declares {entries} entries of {columns} columns, outside what T.800 allows.");
+ if (body.Length < 3 + columns)
+ throw new InvalidDataException("JPEG 2000: the palette box is too short for its column depths.");
+
+ var depths = new int[columns];
+ var bytes = new int[columns];
+ var rowBytes = 0;
+ for (var c = 0; c < columns; c++)
+ {
+ var b = body[3 + c];
+ if ((b & 0x80) != 0)
+ throw new NotSupportedException("JPEG 2000: signed palette columns are not implemented.");
+
+ depths[c] = (b & 0x7F) + 1;
+ if (depths[c] > 16)
+ throw new NotSupportedException(
+ $"JPEG 2000: a {depths[c]}-bit palette column is not implemented; the limit here is 16.");
+ bytes[c] = (depths[c] + 7) / 8;
+ rowBytes += bytes[c];
+ }
+
+ var offset = 3 + columns;
+ if (body.Length - offset < (long)entries * rowBytes)
+ throw new InvalidDataException("JPEG 2000: the palette box is too short for its entries.");
+
+ var values = new ushort[entries * columns];
+ for (var e = 0; e < entries; e++)
+ {
+ for (var c = 0; c < columns; c++)
+ {
+ int value = bytes[c] == 1 ? body[offset] : BinaryPrimitives.ReadUInt16BigEndian(body[offset..]);
+ offset += bytes[c];
+
+ // Bits above the column's declared depth are not part of the value.
+ values[e * columns + c] = (ushort)(value & ((1 << depths[c]) - 1));
+ }
+ }
+
+ return new Jp2ChannelMapping(entries, columns, depths, values, []);
+ }
+
+ ///
+ /// The component mapping box (I.5.3.5): four bytes per output channel, CMP(2) MTYP(1) PCOL(1).
+ /// MTYP 0 takes component CMP as it is, MTYP 1 looks it up in palette column PCOL.
+ ///
+ private static (int Component, int Column)[] ReadComponentMapping(ReadOnlySpan body)
+ {
+ if (body.Length == 0 || body.Length % 4 != 0)
+ throw new InvalidDataException("JPEG 2000: the component mapping box is not a whole number of entries.");
+
+ var map = new (int Component, int Column)[body.Length / 4];
+ for (var i = 0; i < map.Length; i++)
+ {
+ var entry = body.Slice(i * 4, 4);
+ int component = BinaryPrimitives.ReadUInt16BigEndian(entry);
+ map[i] = entry[2] switch
+ {
+ 0 => (component, -1),
+ 1 => (component, entry[3]),
+ _ => throw new InvalidDataException(
+ $"JPEG 2000: the component mapping box has mapping type {entry[2]}, which T.800 does not define."),
+ };
+ }
+
+ return map;
}
///
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/Jpeg2000Decoder.cs b/src/Sdcb.SimdCodecs.Jpeg2000/Jpeg2000Decoder.cs
index 02c6e3c..b568c66 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/Jpeg2000Decoder.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/Jpeg2000Decoder.cs
@@ -1,4 +1,6 @@
using System;
+using System.Buffers;
+using System.Collections.Generic;
using System.IO;
using System.Threading;
@@ -100,15 +102,23 @@ public sealed record Jpeg2000Info(
/// 15444-1, Part 1).
///
/// What this decodes today. A raw J2K codestream, or one in a JP2 file
-/// with no palette, component mapping or channel definition, holding any number of
-/// 1-to-16-bit unsigned components of one precision, in a single tile, at full
-/// resolution, coded in any number of quality layers with maximal precincts and
-/// LRCP progression: losslessly with the reversible 5/3 wavelet, or lossily with
-/// the irreversible 9/7 wavelet and scalar quantization, with or without the
-/// matching component transform (RCT or ICT). Quality layers are supported
-/// because every /JPXDecode image measured in real PDFs has several.
-/// Anything outside it raises naming the
-/// feature and the rung that owns it — never a plausible-looking wrong raster.
+/// (with or without a palette and component mapping, but no channel definition),
+/// holding any number of 1-to-16-bit unsigned components of one precision, in any
+/// number of tiles and tile-parts, coded in any number of quality layers, with any
+/// precinct sizes, in any of the five progression orders, with or without SOP and
+/// EPH markers: losslessly with the reversible 5/3 wavelet, or lossily with the
+/// irreversible 9/7 wavelet and scalar quantization, with or without the matching
+/// component transform (RCT or ICT). Of the code-block styles, context reset,
+/// predictable termination and segmentation symbols are decoded. Anything outside
+/// it (POC, PPM / PPT, RGN, tile-part header overrides, the bypass, per-pass
+/// termination and vertically causal code-block styles, subsampling, signed or
+/// mixed-precision components) raises naming
+/// the feature — never a plausible-looking wrong raster.
+///
+///
+/// Issue #1's JPEG 2000 files are the real-world check on all of that; see
+/// Issue1Jpeg2000CorpusTests in the test project for how to run them after
+/// changing this decoder.
///
///
/// Why there is no partial-credit mode. JPEG 2000 has no early pipeline
@@ -171,10 +181,9 @@ public static Jpeg2000Image Decode(ReadOnlySpan data, int reduce, Cancella
if (!Jp2Reader.LooksLikeJp2(data)) return DecodeCodestream(data, reduce, cancellationToken);
var layout = Jp2Reader.Read(data);
- return DecodeCodestream(data.Slice(layout.CodestreamStart, layout.CodestreamLength), reduce, cancellationToken) with
- {
- Colour = layout.Colour,
- };
+ var image = DecodeCodestream(data.Slice(layout.CodestreamStart, layout.CodestreamLength), reduce, cancellationToken);
+ if (layout.Channels is { } channels) image = ApplyChannels(image, channels);
+ return image with { Colour = layout.Colour };
}
///
@@ -188,11 +197,13 @@ public static Jpeg2000Info ReadInfo(ReadOnlySpan data)
{
var codestream = data;
Jp2Colour? colour = null;
+ Jp2ChannelMapping? channels = null;
if (Jp2Reader.LooksLikeJp2(data))
{
var layout = Jp2Reader.Read(data);
codestream = data.Slice(layout.CodestreamStart, layout.CodestreamLength);
colour = layout.Colour;
+ channels = layout.Channels;
}
var header = CodestreamReader.Read(codestream);
@@ -200,7 +211,15 @@ public static Jpeg2000Info ReadInfo(ReadOnlySpan data)
var levels = int.MaxValue;
foreach (var coding in header.ComponentCoding) levels = Math.Min(levels, coding.DecompositionLevels);
- return new Jpeg2000Info(siz.Width, siz.Height, siz.Components.Length, siz.Components[0].BitDepth, levels, colour)
+ var components = siz.Components.Length;
+ var depth = siz.Components[0].BitDepth;
+ if (channels is not null)
+ {
+ depth = channels.OutputDepth(ComponentDepths(components, depth));
+ components = channels.Map.Length;
+ }
+
+ return new Jpeg2000Info(siz.Width, siz.Height, components, depth, levels, colour)
{
Region = new Rect(siz.X0, siz.Y0, siz.X1, siz.Y1),
};
@@ -228,6 +247,42 @@ public static int ReductionFor(int width, int height, int minLongEdge)
return reduce;
}
+ private static int[] ComponentDepths(int components, int depth)
+ {
+ var depths = new int[components];
+ Array.Fill(depths, depth);
+ return depths;
+ }
+
+ ///
+ /// The JP2 component mapping (T.800 I.5.3.5) applied to the decoded components: each output
+ /// channel is either a component as it is or a palette column looked up by a component's sample.
+ /// An index past the last palette entry takes the last entry, which only a damaged file needs.
+ ///
+ private static Jpeg2000Image ApplyChannels(Jpeg2000Image image, Jp2ChannelMapping channels)
+ {
+ var depth = channels.OutputDepth(ComponentDepths(image.Components, image.BitDepth));
+ var map = channels.Map;
+ var pixels = (long)image.Width * image.Height;
+ var output = new ushort[pixels * map.Length];
+
+ for (var i = 0; i < map.Length; i++)
+ {
+ var (component, column) = map[i];
+ var source = image.Samples;
+ var stride = image.Components;
+ for (long p = 0, o = i, s = component; p < pixels; p++, o += map.Length, s += stride)
+ {
+ var value = source[s];
+ output[o] = column < 0
+ ? value
+ : channels.Values[Math.Min(value, channels.Entries - 1) * channels.Columns + column];
+ }
+ }
+
+ return new Jpeg2000Image(image.Width, image.Height, depth, output, map.Length);
+ }
+
private static Jpeg2000Image DecodeCodestream(ReadOnlySpan data, int reduce, CancellationToken cancellationToken = default)
{
var header = CodestreamReader.Read(data);
@@ -239,70 +294,179 @@ private static Jpeg2000Image DecodeCodestream(ReadOnlySpan data, int reduc
foreach (var coding in header.ComponentCoding) levels = Math.Min(levels, coding.DecompositionLevels);
reduce = Math.Min(reduce, levels);
+ var tiles = GroupTileParts(header);
+ var componentCount = siz.Components.Length;
+
+ // The decoded image's region on the reference grid at this reduction (Equation B-14): the
+ // tiles' reduced regions, rounded the same way, cover it exactly.
+ var region = new Rect(
+ TileComponent.CeilDivPow2(siz.X0, reduce), TileComponent.CeilDivPow2(siz.Y0, reduce),
+ TileComponent.CeilDivPow2(siz.X1, reduce), TileComponent.CeilDivPow2(siz.Y1, reduce));
+ var total = region.Area * componentCount;
+ if (total > Array.MaxLength)
+ throw new InvalidDataException(
+ $"JPEG 2000: a {region.Width}x{region.Height} image of {componentCount} components is " +
+ $"{total:N0} samples, more than one array can hold.");
+
+ // Allocated once and written tile by tile, so a decode holds the output and one tile's
+ // working set at a time. A codestream must carry at least one tile-part per tile (checked
+ // above), so the tile count, and with it this loop, is bounded by the data itself.
+ var output = new ushort[total];
+ var blockState = new BlockState();
+ var tilesWide = (int)siz.TilesWide;
+
+ for (var t = 0; t < tiles.Length; t++)
+ {
+ cancellationToken.ThrowIfCancellationRequested();
+
+ // T.800 Equation B-7: the tile's extent, clipped to the image region.
+ var p = t % tilesWide;
+ var q = t / tilesWide;
+ var tile = new Rect(
+ (int)Math.Max(siz.TileX0 + (long)p * siz.TileWidth, siz.X0),
+ (int)Math.Max(siz.TileY0 + (long)q * siz.TileHeight, siz.Y0),
+ (int)Math.Min(siz.TileX0 + (long)(p + 1) * siz.TileWidth, siz.X1),
+ (int)Math.Min(siz.TileY0 + (long)(q + 1) * siz.TileHeight, siz.Y1));
+
+ DecodeTile(header, data, tiles[t], tile, reduce, region, output, blockState, cancellationToken);
+ }
+
+ return new Jpeg2000Image(region.Width, region.Height, siz.Components[0].BitDepth, output, componentCount);
+ }
+
+ ///
+ /// Every tile's tile-parts, in TPsot order. T.800 A.4.2 lets the parts of different tiles
+ /// interleave, but each tile's own parts carry its packets in sequence, and every tile has at
+ /// least one.
+ ///
+ private static List[] GroupTileParts(CodestreamHeader header)
+ {
+ var count = (int)header.Siz.TileCount;
+ if (header.TileParts.Count < count)
+ throw new InvalidDataException(
+ $"JPEG 2000: SIZ declares {count:N0} tiles, but the codestream has only " +
+ $"{header.TileParts.Count:N0} tile-parts.");
+
+ var tiles = new List[count];
+ foreach (var part in header.TileParts) (tiles[part.TileIndex] ??= []).Add(part);
+
+ for (var t = 0; t < count; t++)
+ {
+ var parts = tiles[t]
+ ?? throw new InvalidDataException($"JPEG 2000: tile {t} has no tile-part.");
+ parts.Sort(static (a, b) => a.PartIndex.CompareTo(b.PartIndex));
+ for (var i = 0; i < parts.Count; i++)
+ {
+ if (parts[i].PartIndex != i)
+ throw new InvalidDataException(
+ $"JPEG 2000: tile {t}'s tile-parts are numbered {parts[i].PartIndex} where {i} was due.");
+ }
+ }
+
+ return tiles;
+ }
+
+ /// One tile, from its packets to its samples in .
+ private static void DecodeTile(
+ CodestreamHeader header, ReadOnlySpan data, List parts, Rect tile, int reduce, Rect region,
+ ushort[] output, BlockState blockState, CancellationToken cancellationToken)
+ {
+ var siz = header.Siz;
+
+ // Each tile is metered against its own extent: the budget bounds amplification, which is
+ // per tile, and a tile's buffers are released before the next one is built.
var budget = new Jpeg2000SampleBudget(
- Jpeg2000Limits.BudgetFor(siz.Width, siz.Height, siz.Components.Length));
+ Jpeg2000Limits.BudgetFor(tile.Width, tile.Height, siz.Components.Length));
var components = new TileComponent[siz.Components.Length];
for (var c = 0; c < components.Length; c++)
{
cancellationToken.ThrowIfCancellationRequested();
components[c] = TileComponent.Build(
- header, c, budget, header.ComponentCoding[c].ResolutionCount - reduce);
+ header, c, tile, budget, header.ComponentCoding[c].ResolutionCount - reduce);
}
- cancellationToken.ThrowIfCancellationRequested();
- var part = header.TileParts[0];
- var tilePartData = data.Slice(part.Start, part.Length);
+ // The packets run on from one tile-part into the next, so a tile split into parts is read
+ // as their bytes put back together.
+ byte[]? rented = null;
+ ReadOnlySpan tileData;
+ if (parts.Count == 1)
+ {
+ tileData = data.Slice(parts[0].Start, parts[0].Length);
+ }
+ else
+ {
+ var length = 0;
+ foreach (var part in parts) length += part.Length;
+ rented = ArrayPool.Shared.Rent(length);
+ var offset = 0;
+ foreach (var part in parts)
+ {
+ data.Slice(part.Start, part.Length).CopyTo(rented.AsSpan(offset));
+ offset += part.Length;
+ }
- // Tier-2 first, over the whole tile-part: it discovers which code-blocks
- // carry data and where, without decoding any of it. The components'
- // packets are interleaved in the tile-part, so it reads them all at once.
- Tier2.ReadPackets(header, components, tilePartData);
+ tileData = rented.AsSpan(0, length);
+ }
- // Then tier-1, block by block, and the inverse wavelet, component by
- // component: in integers for a 5/3 component, in floats for a 9/7 one.
- var integers = new int[components.Length][];
- var reals = new float[components.Length][];
- var blockState = new BlockState();
- for (var c = 0; c < components.Length; c++)
+ try
{
- cancellationToken.ThrowIfCancellationRequested();
- var tile = components[c];
- for (var r = 0; r < tile.DecodedResolutions; r++)
+ // Tier-2 first, over the whole tile: it discovers which code-blocks carry data and
+ // where, without decoding any of it. The components' packets are interleaved, so it
+ // reads them all at once.
+ Tier2.ReadPackets(header, components, tileData, tile);
+
+ // Then tier-1, block by block, and the inverse wavelet, component by component: in
+ // integers for a 5/3 component, in floats for a 9/7 one.
+ var integers = new int[components.Length][];
+ var reals = new float[components.Length][];
+ for (var c = 0; c < components.Length; c++)
{
- foreach (var band in tile.Resolutions[r].Bands)
+ cancellationToken.ThrowIfCancellationRequested();
+ var tileComponent = components[c];
+ for (var r = 0; r < tileComponent.DecodedResolutions; r++)
{
- foreach (var block in band.Blocks)
+ foreach (var band in tileComponent.Resolutions[r].Bands)
{
- BlockDecoder.Decode(band, block, tilePartData, blockState);
+ foreach (var block in band.Blocks)
+ {
+ BlockDecoder.Decode(band, block, tileData, blockState);
+ }
}
}
+
+ var decodedBounds = tileComponent.Resolutions[tileComponent.DecodedResolutions - 1].Bounds;
+ budget.Charge(decodedBounds.Width, decodedBounds.Height);
+ if (tileComponent.Irreversible)
+ reals[c] = InverseWavelet.ReconstructIrreversible(tileComponent, tileComponent.DecodedResolutions);
+ else
+ integers[c] = InverseWavelet.Reconstruct(tileComponent, tileComponent.DecodedResolutions);
}
- var decodedBounds = tile.Resolutions[tile.DecodedResolutions - 1].Bounds;
- budget.Charge(decodedBounds.Width, decodedBounds.Height);
- if (tile.Irreversible) reals[c] = InverseWavelet.ReconstructIrreversible(tile, tile.DecodedResolutions);
- else integers[c] = InverseWavelet.Reconstruct(tile, tile.DecodedResolutions);
- }
+ // G.1.2: the component transform comes between the wavelet and the level shift. The
+ // reader has already checked that components 0 to 2 share one wavelet, which picks RCT
+ // for the 5/3 and ICT for the 9/7.
+ if (header.MultipleComponentTransform)
+ {
+ if (components[0].Irreversible) ComponentTransform.InverseIct(reals[0], reals[1], reals[2]);
+ else ComponentTransform.InverseRct(integers[0], integers[1], integers[2]);
+ }
- // G.1.2: the component transform comes between the wavelet and the
- // level shift. The reader has already checked that components 0 to 2
- // share one wavelet, which picks RCT for the 5/3 and ICT for the 9/7.
- if (header.MultipleComponentTransform)
+ var first = components[0];
+ LevelShift(integers, reals, first.Resolutions[first.DecodedResolutions - 1].Bounds,
+ siz.Components[0].BitDepth, output, region);
+ }
+ finally
{
- if (components[0].Irreversible) ComponentTransform.InverseIct(reals[0], reals[1], reals[2]);
- else ComponentTransform.InverseRct(integers[0], integers[1], integers[2]);
+ if (rented is not null) ArrayPool.Shared.Return(rented);
}
-
- var first = components[0];
- return LevelShift(
- integers, reals, first.Resolutions[first.DecodedResolutions - 1].Bounds, siz.Components[0].BitDepth);
}
///
/// T.800 G.1.2: undo the DC level shift the encoder applied to make an
/// unsigned component signed, then clamp to the declared precision, and
- /// interleave the components into one sample per component per pixel.
+ /// interleave the components into one sample per component per pixel, at the
+ /// tile's place in the image.
///
/// The clamp is not belt-and-braces. A lossless codestream reconstructs
/// exactly and needs none, but a truncated or hostile one can produce
@@ -318,33 +482,38 @@ private static Jpeg2000Image DecodeCodestream(ReadOnlySpan data, int reduc
///
/// Each reversible component's samples, null for an irreversible one.
/// Each irreversible component's samples, null for a reversible one.
- private static Jpeg2000Image LevelShift(int[]?[] integers, float[]?[] reals, Rect bounds, int bitDepth)
+ /// The tile's decoded extent, on the same grid as .
+ /// The whole image's interleaved samples.
+ /// The whole image's decoded extent.
+ private static void LevelShift(
+ int[]?[] integers, float[]?[] reals, Rect bounds, int bitDepth, ushort[] output, Rect region)
{
var shift = 1 << (bitDepth - 1);
var maximum = (1 << bitDepth) - 1;
var count = integers.Length;
+ var width = bounds.Width;
- var output = new ushort[(long)bounds.Width * bounds.Height * count];
for (var c = 0; c < count; c++)
{
- if (integers[c] is { } plane)
+ for (var y = 0; y < bounds.Height; y++)
{
- for (int i = 0, o = c; i < plane.Length; i++, o += count)
+ var o = (((long)(bounds.Y0 - region.Y0 + y) * region.Width) + (bounds.X0 - region.X0)) * count + c;
+ var i = y * width;
+ if (integers[c] is { } plane)
{
- output[o] = (ushort)Math.Clamp(plane[i] + shift, 0, maximum);
+ for (var x = 0; x < width; x++, i++, o += count)
+ output[o] = (ushort)Math.Clamp(plane[i] + shift, 0, maximum);
}
- }
- else
- {
- var real = reals[c]!;
- for (int i = 0, o = c; i < real.Length; i++, o += count)
+ else
{
- // Clamped as a float, so a value no int can hold still lands on a rail.
- output[o] = (ushort)Math.Clamp(MathF.Round(real[i]) + shift, 0f, maximum);
+ var real = reals[c]!;
+ for (var x = 0; x < width; x++, i++, o += count)
+ {
+ // Clamped as a float, so a value no int can hold still lands on a rail.
+ output[o] = (ushort)Math.Clamp(MathF.Round(real[i]) + shift, 0f, maximum);
+ }
}
}
}
-
- return new Jpeg2000Image(bounds.Width, bounds.Height, bitDepth, output, count);
}
}
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/Jpeg2000ImageDecoder.cs b/src/Sdcb.SimdCodecs.Jpeg2000/Jpeg2000ImageDecoder.cs
index 5644b1f..611a528 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/Jpeg2000ImageDecoder.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/Jpeg2000ImageDecoder.cs
@@ -61,20 +61,13 @@ public static bool TryReadInfo(ReadOnlySpan data, out ImageInfo info)
try
{
- var codestream = data;
- Jp2Colour? colour = null;
- if (Jpeg2000Decoder.IsJp2(data))
- {
- var layout = Jp2Reader.Read(data);
- codestream = data.Slice(layout.CodestreamStart, layout.CodestreamLength);
- colour = layout.Colour;
- }
-
- var siz = CodestreamReader.Read(codestream).Siz;
- if (!Presentable(siz.Components.Length, colour)) return false;
+ // ReadInfo, not SIZ alone: a JP2 palette changes how many channels, and of what depth,
+ // the decoded image has.
+ var j2k = Jpeg2000Decoder.ReadInfo(data);
+ if (!Presentable(j2k.Components, j2k.Colour)) return false;
- var depth = siz.Components[0].BitDepth;
- info = new ImageInfo(siz.Width, siz.Height, siz.Components.Length, FormatFor(depth))
+ var depth = j2k.BitDepth;
+ info = new ImageInfo(j2k.Width, j2k.Height, j2k.Components, FormatFor(depth))
{
Format = ImageFormat.Jpeg2000,
BitDepth = depth,
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/Markers.cs b/src/Sdcb.SimdCodecs.Jpeg2000/Markers.cs
index 29d709f..afb675e 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/Markers.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/Markers.cs
@@ -209,6 +209,28 @@ public int PrecinctHeightExponent(int resolution) =>
PrecinctSizes.Length == 0 ? 15 : (PrecinctSizes[Math.Min(resolution, PrecinctSizes.Length - 1)] >> 4) & 0x0F;
}
+/// The code-block style flags of T.800 Table A.19.
+internal static class CodeBlockStyles
+{
+ /// Selective arithmetic coding bypass.
+ public const int Bypass = 0x01;
+
+ /// Reset the context probabilities at the end of every coding pass.
+ public const int ResetContexts = 0x02;
+
+ /// Terminate the codeword at the end of every coding pass.
+ public const int TerminateEachPass = 0x04;
+
+ /// Vertically causal context formation.
+ public const int VerticallyCausal = 0x08;
+
+ /// Predictable termination: a constraint on the encoder, invisible to a decoder.
+ public const int PredictableTermination = 0x10;
+
+ /// A four-symbol segmentation marker after every cleanup pass.
+ public const int SegmentationSymbols = 0x20;
+}
+
/// Quantization style (T.800 Table A.28), the low five bits of Sqcd.
internal enum QuantizationStyle
{
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/PacketBitReader.cs b/src/Sdcb.SimdCodecs.Jpeg2000/PacketBitReader.cs
index cbc7a5f..6708a96 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/PacketBitReader.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/PacketBitReader.cs
@@ -89,9 +89,9 @@ public void AlignToByte()
/// Hazard 5. That kept state spans the packets of successive quality
/// layers, not just the leaves within one packet. Re-creating the trees per
/// packet decodes layer 0 correctly and every later layer wrongly, and a
-/// single-layer fixture cannot tell the difference — which is why rung 1 owning
-/// this class does not mean rung 1 has tested it. That is rung 3's job, and it
-/// needs a multi-layer fixture built for the purpose.
+/// single-layer fixture cannot tell the difference, which is why the multi-layer
+/// fixtures exist. The trees belong to a precinct ()
+/// and live as long as it does.
///
///
internal sealed class TagTree
diff --git a/src/Sdcb.SimdCodecs.Jpeg2000/Tier2.cs b/src/Sdcb.SimdCodecs.Jpeg2000/Tier2.cs
index 9e6ad69..1be6a6c 100644
--- a/src/Sdcb.SimdCodecs.Jpeg2000/Tier2.cs
+++ b/src/Sdcb.SimdCodecs.Jpeg2000/Tier2.cs
@@ -5,64 +5,184 @@
namespace Sdcb.SimdCodecs.Jpeg2000;
///
-/// Tier-2: the packet layer (T.800 B.9 and B.10). It reads packet headers to
+/// Tier-2: the packet layer (T.800 B.9 to B.12). It reads packet headers to
/// discover which code-blocks carry data, how many coding passes each
/// got and how many bytes those passes occupy, then hands the byte ranges to
/// tier-1. It never touches a coefficient.
///
-/// The envelope keeps the packet sequence simple: one tile, and maximal precincts
-/// so each resolution of each component is a single precinct. Under LRCP that is
-/// one packet per layer, resolution and component, nested in that order. The five
-/// progression orders differ only in how they nest those loops, which is why
-/// still refuses the other four rather than
-/// accepting them for free: with several layers and components they really do
-/// visit the packets in different sequences, and only this one has a fixture.
+/// A tile's packets are one per layer, resolution, component and precinct, and
+/// the progression order of COD says how those four loops nest (B.12.1). The two
+/// layer-and-resolution orders, LRCP and RLCP, walk precincts by their index. The
+/// three position-driven ones, RPCL, PCRL and CPRL, walk the reference grid and
+/// take each precinct at the first grid point that lands in it; that point is
+/// worked out once per precinct (), so the walk
+/// becomes a sort by position rather than a visit of every grid point.
///
///
/// Layers are where tier-2 becomes stateful. A code-block's inclusion,
/// zero bit-planes, Lblock and pass count, and its precinct's tag trees,
/// all carry over from one layer's packet to the next. A decoder that rebuilt any
-/// of them per packet would read layer 0 correctly and every later layer wrongly,
-/// which is hazard 5 of the roadmap and the reason the multi-layer fixtures exist.
+/// of them per packet would read layer 0 correctly and every later layer wrongly.
///
///
internal static class Tier2
{
///
- /// Reads every packet of the tile-part, populating each code-block's
- /// inclusion state and coded byte ranges.
+ /// Reads every packet of a single-tile codestream's one tile. See
+ /// .
+ ///
+ public static void ReadPackets(CodestreamHeader header, TileComponent[] components, ReadOnlySpan data)
+ {
+ var siz = header.Siz;
+ ReadPackets(header, components, data, new Rect(siz.X0, siz.Y0, siz.X1, siz.Y1));
+ }
+
+ ///
+ /// Reads every packet of one tile, populating each code-block's inclusion state and coded byte
+ /// ranges.
///
/// The parsed main header.
/// Each component's tile-component geometry to fill in.
- /// The tile-part's coded data, starting just after SOD.
- public static void ReadPackets(CodestreamHeader header, TileComponent[] components, ReadOnlySpan data)
+ /// The tile's coded data: every tile-part's bytes after SOD, in order.
+ /// The tile's extent on the reference grid, unused here but kept for the caller's symmetry.
+ public static void ReadPackets(CodestreamHeader header, TileComponent[] components, ReadOnlySpan data, Rect tile)
{
- var offset = 0;
+ _ = tile;
+ var reader = new PacketStream(header.UseSopMarkers, header.UseEphMarkers);
+ var layers = header.Layers;
var resolutions = 0;
foreach (var component in components) resolutions = Math.Max(resolutions, component.Resolutions.Length);
- // B.12.1.1, LRCP: layer, then resolution, then component, then precinct, the last of which is
- // length 1 here. A COC can give components different numbers of decomposition levels, and a
- // component simply has no packet at a resolution it does not have.
- for (var layer = 0; layer < header.Layers; layer++)
+ // A COC can give components different numbers of decomposition levels, and a component
+ // simply has no packet at a resolution it does not have.
+ switch (header.Progression)
{
- for (var r = 0; r < resolutions; r++)
- {
- foreach (var component in components)
+ case ProgressionOrder.Lrcp:
+ // B.12.1.1: layer, resolution, component, precinct.
+ for (var layer = 0; layer < layers; layer++)
+ for (var r = 0; r < resolutions; r++)
+ foreach (var component in components)
+ {
+ if (r >= component.Resolutions.Length) continue;
+ foreach (var precinct in component.Resolutions[r].Precincts)
+ reader.Read(precinct, data, layer);
+ }
+ break;
+
+ case ProgressionOrder.Rlcp:
+ // B.12.1.2: resolution, layer, component, precinct.
+ for (var r = 0; r < resolutions; r++)
+ for (var layer = 0; layer < layers; layer++)
+ foreach (var component in components)
+ {
+ if (r >= component.Resolutions.Length) continue;
+ foreach (var precinct in component.Resolutions[r].Precincts)
+ reader.Read(precinct, data, layer);
+ }
+ break;
+
+ case ProgressionOrder.Rpcl:
+ // B.12.1.3: resolution, then position (y, x), then component, then layer.
+ for (var r = 0; r < resolutions; r++)
{
- if (r >= component.Resolutions.Length) continue;
- offset = ReadPacket(component.Resolutions[r], data, offset, layer);
+ var visits = new List();
+ for (var c = 0; c < components.Length; c++)
+ if (r < components[c].Resolutions.Length)
+ AddVisits(visits, components[c].Resolutions[r], c, r);
+
+ visits.Sort(static (a, b) => Compare(a, b, componentFirst: true));
+ ReadAllLayers(ref reader, visits, data, layers);
}
+ break;
+
+ case ProgressionOrder.Pcrl:
+ {
+ // B.12.1.4: position (y, x), then component, then resolution, then layer.
+ var visits = new List();
+ for (var c = 0; c < components.Length; c++)
+ for (var r = 0; r < components[c].Resolutions.Length; r++)
+ AddVisits(visits, components[c].Resolutions[r], c, r);
+
+ visits.Sort(static (a, b) => Compare(a, b, componentFirst: true));
+ ReadAllLayers(ref reader, visits, data, layers);
+ break;
}
+
+ case ProgressionOrder.Cprl:
+ // B.12.1.5: component, then position (y, x), then resolution, then layer.
+ for (var c = 0; c < components.Length; c++)
+ {
+ var visits = new List();
+ for (var r = 0; r < components[c].Resolutions.Length; r++)
+ AddVisits(visits, components[c].Resolutions[r], c, r);
+
+ visits.Sort(static (a, b) => Compare(a, b, componentFirst: false));
+ ReadAllLayers(ref reader, visits, data, layers);
+ }
+ break;
+
+ default:
+ throw new InvalidDataException($"JPEG 2000: progression order {header.Progression} is not defined.");
}
}
- private static int ReadPacket(Resolution resolution, ReadOnlySpan data, int offset, int layer)
+ /// One precinct, with what the position-driven orders sort it by.
+ private readonly record struct Visit(Precinct Precinct, long Y, long X, int Component, int Resolution);
+
+ private static void AddVisits(List visits, Resolution resolution, int component, int r)
+ {
+ foreach (var precinct in resolution.Precincts)
+ visits.Add(new Visit(precinct, precinct.VisitY, precinct.VisitX, component, r));
+ }
+
+ ///
+ /// The position orders sort by y, then x, then the two inner loops: component before resolution
+ /// for RPCL (where the resolution is fixed anyway) and PCRL, resolution alone for CPRL (where the
+ /// component is). Two precincts of the same component and resolution never share a visiting
+ /// point, so the order is total.
+ ///
+ private static int Compare(Visit a, Visit b, bool componentFirst)
+ {
+ var order = a.Y.CompareTo(b.Y);
+ if (order != 0) return order;
+ order = a.X.CompareTo(b.X);
+ if (order != 0) return order;
+ if (componentFirst)
+ {
+ order = a.Component.CompareTo(b.Component);
+ return order != 0 ? order : a.Resolution.CompareTo(b.Resolution);
+ }
+
+ return a.Resolution.CompareTo(b.Resolution);
+ }
+
+ private static void ReadAllLayers(ref PacketStream reader, List visits, ReadOnlySpan data, int layers)
+ {
+ foreach (var visit in visits)
+ for (var layer = 0; layer < layers; layer++)
+ reader.Read(visit.Precinct, data, layer);
+ }
+
+ /// The read position through a tile's packets, and the two optional markers around each.
+ private struct PacketStream(bool sop, bool eph)
+ {
+ private int _offset;
+
+ public void Read(Precinct precinct, ReadOnlySpan data, int layer) =>
+ _offset = ReadPacket(precinct, data, _offset, layer, sop, eph);
+ }
+
+ private static int ReadPacket(Precinct precinct, ReadOnlySpan data, int offset, int layer, bool sop, bool eph)
{
if (offset > data.Length)
throw new InvalidDataException("JPEG 2000: ran out of tile-part data before the last packet.");
+ // A.8.1: when COD allows SOP markers, each packet MAY start with one: FF91, a length of 4
+ // and a 16-bit sequence number. Nothing in it changes how the packet is read.
+ if (sop && offset + 6 <= data.Length && data[offset] == 0xFF && data[offset + 1] == 0x91)
+ offset += 6;
+
var reader = new PacketBitReader(data[offset..]);
// B.10.3: a leading zero says the packet is empty. It is not a corner
@@ -71,25 +191,24 @@ private static int ReadPacket(Resolution resolution, ReadOnlySpan data, in
if (reader.ReadBit() == 0)
{
reader.AlignToByte();
- return offset + reader.Position;
+ return SkipEph(data, offset + reader.Position, eph);
}
var pending = new List<(CodeBlock Block, int Length)>();
- foreach (var band in resolution.Bands)
+ foreach (var part in precinct.Bands)
{
- if (band.Bounds.IsEmpty) continue;
+ if (part.BlocksWide == 0 || part.BlocksHigh == 0) continue;
- // One precinct per resolution, so one pair of tag trees per band,
- // covering the whole code-block grid. Rung 3 makes these per-precinct
- // and must keep them alive across layers -- see TagTree's remarks.
- band.EnsureTagTrees();
+ var band = part.Band;
+ part.EnsureTagTrees();
- for (var by = 0; by < band.BlocksHigh; by++)
+ // B.10: the precinct's code-blocks in each band, in raster order.
+ for (var by = 0; by < part.BlocksHigh; by++)
{
- for (var bx = 0; bx < band.BlocksWide; bx++)
+ for (var bx = 0; bx < part.BlocksWide; bx++)
{
- var block = band.Blocks[by * band.BlocksWide + bx];
+ var block = band.Blocks[(part.BlockY0 + by) * band.BlocksWide + part.BlockX0 + bx];
bool included;
if (block.Included)
@@ -103,7 +222,7 @@ private static int ReadPacket(Resolution resolution, ReadOnlySpan data, in
// Not yet included: the tag tree holds the layer number in
// which it first will be, so "included now" is "that
// number is at most this layer".
- included = band.InclusionTree!.Decode(ref reader, bx, by, layer + 1);
+ included = part.InclusionTree!.Decode(ref reader, bx, by, layer + 1);
}
if (!included) continue;
@@ -118,7 +237,7 @@ private static int ReadPacket(Resolution resolution, ReadOnlySpan data, in
// driven by attacker-supplied bits and must be guaranteed
// to advance.
var threshold = 1;
- while (!band.ZeroBitPlaneTree!.Decode(ref reader, bx, by, threshold))
+ while (!part.ZeroBitPlaneTree!.Decode(ref reader, bx, by, threshold))
{
if (++threshold > MaxZeroBitPlanes)
throw new InvalidDataException(
@@ -126,7 +245,7 @@ private static int ReadPacket(Resolution resolution, ReadOnlySpan data, in
"all-zero bit-planes, which no legal codestream does.");
}
- block.ZeroBitPlanes = band.ZeroBitPlaneTree!.Value(bx, by);
+ block.ZeroBitPlanes = part.ZeroBitPlaneTree!.Value(bx, by);
}
var newPasses = ReadPassCount(ref reader);
@@ -160,7 +279,7 @@ private static int ReadPacket(Resolution resolution, ReadOnlySpan data, in
// The bodies follow the header in the same order the header described
// them, back to back with nothing between.
- var bodyOffset = offset + reader.Position;
+ var bodyOffset = SkipEph(data, offset + reader.Position, eph);
foreach (var (block, length) in pending)
{
if (length > data.Length - bodyOffset)
@@ -182,6 +301,13 @@ private static int ReadPacket(Resolution resolution, ReadOnlySpan data, in
///
private const int MaxZeroBitPlanes = 74;
+ ///
+ /// A.8.2: when COD allows EPH markers, FF92 may follow each packet header, before the bodies.
+ /// Skipped when present; a header without one is read as though it were not allowed.
+ ///
+ private static int SkipEph(ReadOnlySpan data, int offset, bool eph) =>
+ eph && offset + 2 <= data.Length && data[offset] == 0xFF && data[offset + 1] == 0x92 ? offset + 2 : offset;
+
/// Lblock starts at 3 and only grows; 32 already makes the length field unreadable.
private const int MaxLBlock = 32;
diff --git a/tests/Sdcb.SimdCodecs.Tests/Issue1Jbig2CorpusTests.cs b/tests/Sdcb.SimdCodecs.Tests/Issue1Jbig2CorpusTests.cs
new file mode 100644
index 0000000..bcf8ff5
--- /dev/null
+++ b/tests/Sdcb.SimdCodecs.Tests/Issue1Jbig2CorpusTests.cs
@@ -0,0 +1,148 @@
+using Sdcb.SimdCodecs.Jbig2;
+using Shouldly;
+
+namespace Sdcb.SimdCodecs.Tests;
+
+///
+/// The JBIG2 files attached to issue #1 (https://github.com/sdcb/SimdCodecs/issues/1),
+/// every page of every one checked against jbig2dec.
+///
+/// The files are not in the repository and must not be. The reporter took them
+/// from a test set whose source and licence they no longer know, and AGENTS.md admits
+/// no third-party data of uncertain provenance, not even as a fetch-by-URL fixture.
+/// So this test runs only when SIMDCODECS_ISSUE1_DIR names a directory holding
+/// them, and skips otherwise (REQUIRE_ORACLES does not change that: this is an optional
+/// corpus, not an oracle).
+///
+///
+/// After changing the JBIG2 decoder, run it. Download
+/// https://github.com/user-attachments/files/34160416/JPX_JBig2.zip (and, for the JPEG 2000
+/// side, https://github.com/user-attachments/files/34161061/JPXDecode.zip), extract them
+/// somewhere outside the repository, set SIMDCODECS_ISSUE1_DIR to that directory
+/// (it is searched recursively for *.jb2), and run
+/// --filter "FullyQualifiedName~Issue1Jbig2Corpus" with jbig2dec available.
+///
+///
+/// Expected: all 114 files decode to jbig2dec's pages, except two. 042_13.jb2 and
+/// 042_14.jb2 are the same page as the other 042_* files, coded with a
+/// refining symbol dictionary (arithmetic and Huffman respectively), and jbig2dec rejects
+/// both at the first refined symbol of their second dictionary ("OOB in single
+/// refinement/aggregate coded symbol data", "invalid number of symbols in aggregate
+/// glyph"). This decoder, written independently from T.88, reads the same fields at the
+/// same place and finds them malformed too: the first refinement of the arithmetic one
+/// codes an out-of-band IARDX, which §6.5.8.2.2 does not allow. Two independent
+/// readings agreeing that a stream is invalid is the best evidence available, so for
+/// those two the expectation is a clean . If a later
+/// change makes them decode, they must decode to the page the other 042_* files
+/// give (jbig2dec's output for 042_1.jb2).
+///
+///
+/// Three files (a.jb2, test.jb2, XeroxWorkCentre5230A_Huffman.jb2)
+/// are PDF-embedded streams with no file header; they go through
+///
+/// with the page size jbig2dec reports for them.
+///
+///
+public sealed class Issue1Jbig2CorpusTests(ITestOutputHelper output)
+{
+ private static readonly string[] RejectedByBoth = ["042_13", "042_14"];
+
+ [Fact]
+ public void EveryFile_DecodesToJbig2decsPages()
+ {
+ var directory = Environment.GetEnvironmentVariable("SIMDCODECS_ISSUE1_DIR");
+ if (string.IsNullOrEmpty(directory))
+ Assert.Skip("SIMDCODECS_ISSUE1_DIR is not set; see Issue1Jbig2CorpusTests for where the issue #1 files come from.");
+ if (!Directory.Exists(directory))
+ Assert.Fail($"SIMDCODECS_ISSUE1_DIR is '{directory}', which is not a directory.");
+
+ Jbig2Oracle.RequireOrSkip();
+ var ct = TestContext.Current.CancellationToken;
+
+ var files = Directory.GetFiles(directory, "*.jb2", SearchOption.AllDirectories).Order().ToArray();
+ files.ShouldNotBeEmpty($"no .jb2 files under {directory}");
+
+ var failures = new List();
+ var byName = files.ToDictionary(f => Path.GetFileNameWithoutExtension(f), StringComparer.OrdinalIgnoreCase);
+
+ foreach (var path in files)
+ {
+ ct.ThrowIfCancellationRequested();
+ var name = Path.GetFileNameWithoutExtension(path);
+ var data = File.ReadAllBytes(path);
+ var isFile = data.AsSpan().StartsWith(Jbig2Decoder.FileSignature);
+
+ try
+ {
+ if (RejectedByBoth.Contains(name))
+ {
+ Jbig2Bitmap[]? ours = null;
+ try { ours = DecodeAll(data, ct); }
+ catch (InvalidDataException) { }
+
+ if (ours is not null)
+ {
+ // Decodes now: then it has to be the page its siblings give.
+ var reference = Jbig2Oracle.DecodePages(File.ReadAllBytes(byName["042_1"]));
+ Compare(name, ours, reference, failures);
+ }
+
+ output.WriteLine($"{name}: {(ours is null ? "rejected as invalid, as jbig2dec does" : "decoded")}");
+ continue;
+ }
+
+ var pages = Jbig2Oracle.DecodePages(data, embedded: !isFile);
+ var decoded = isFile
+ ? DecodeAll(data, ct)
+ : [ToBitmap(Jbig2Decoder.Decode(data, pages[0].Width, pages[0].Height, ct))];
+
+ Compare(name, decoded, pages, failures);
+ output.WriteLine($"{name}: {decoded.Length} page(s) match");
+ }
+ catch (Exception e) when (e is InvalidDataException or NotSupportedException)
+ {
+ failures.Add($"{name}: {e.GetType().Name}: {e.Message}");
+ }
+ }
+
+ failures.ShouldBeEmpty(string.Join('\n', failures));
+ }
+
+ private static Jbig2Bitmap[] DecodeAll(byte[] data, CancellationToken ct)
+ {
+ Jbig2Decoder.TryReadFileInfo(data, out _, out _, out var pages).ShouldBeTrue();
+ var result = new Jbig2Bitmap[Math.Max(1, pages)];
+ for (var p = 0; p < result.Length; p++) result[p] = ToBitmap(Jbig2Decoder.DecodeFile(data, p + 1, ct));
+ return result;
+ }
+
+ private static Jbig2Bitmap ToBitmap(Jbig2Image image)
+ {
+ var bitmap = new Jbig2Bitmap(image.Width, image.Height);
+ image.Bits.CopyTo(bitmap.Data);
+ return bitmap;
+ }
+
+ private static void Compare(string name, Jbig2Bitmap[] ours, IReadOnlyList reference, List failures)
+ {
+ if (ours.Length != reference.Count)
+ {
+ failures.Add($"{name}: {ours.Length} page(s), jbig2dec {reference.Count}");
+ return;
+ }
+
+ for (var p = 0; p < ours.Length; p++)
+ {
+ var (w, h, bits) = (reference[p].Width, reference[p].Height, reference[p].Bits);
+ if (ours[p].Width != w || ours[p].Height != h)
+ {
+ failures.Add($"{name} page {p + 1}: {ours[p].Width}x{ours[p].Height}, jbig2dec {w}x{h}");
+ continue;
+ }
+
+ var differing = 0;
+ for (var i = 0; i < bits.Length; i++) if (bits[i] != ours[p].Data[i]) differing++;
+ if (differing != 0) failures.Add($"{name} page {p + 1}: {differing} pixels differ from jbig2dec");
+ }
+ }
+}
diff --git a/tests/Sdcb.SimdCodecs.Tests/Issue1Jpeg2000CorpusTests.cs b/tests/Sdcb.SimdCodecs.Tests/Issue1Jpeg2000CorpusTests.cs
new file mode 100644
index 0000000..d36c229
--- /dev/null
+++ b/tests/Sdcb.SimdCodecs.Tests/Issue1Jpeg2000CorpusTests.cs
@@ -0,0 +1,98 @@
+using Sdcb.SimdCodecs.Jpeg2000;
+using Shouldly;
+
+namespace Sdcb.SimdCodecs.Tests;
+
+///
+/// The JPEG 2000 files a user attached to issue #1 (https://github.com/sdcb/SimdCodecs/issues/1),
+/// decoded and compared with opj_decompress: reversible images exactly, irreversible ones to
+/// within one, as everywhere else.
+///
+/// The files are not in the repository and must not be added. They came from a test set
+/// whose source and licence nobody could name, so under the clean-room rules in AGENTS.md
+/// they may be run locally but neither committed nor fetched by a test. This test therefore skips
+/// unless SIMDCODECS_ISSUE1_DIR names a directory holding them, and that skip is not an
+/// oracle skip: REQUIRE_ORACLES does not turn it into a failure.
+///
+///
+/// After changing the JPEG 2000 decoder, run it. Download both attachments of issue #1,
+/// https://github.com/user-attachments/files/34160416/JPX_JBig2.zip and
+/// https://github.com/user-attachments/files/34161061/JPXDecode.zip, unzip them into one
+/// directory (the .jb2 files beside them are the JBIG2 half of the same issue), set
+/// SIMDCODECS_ISSUE1_DIR to it and run this class. Its sixteen files cover multiple tiles
+/// and tile-parts, RLCP and RPCL progressions, precincts, SOP and EPH markers, segmentation symbols
+/// and JP2 palettes, as real encoders write them; the committed tests that cover the same ground
+/// are in .
+///
+///
+public class Issue1Jpeg2000CorpusTests
+{
+ /// The environment variable naming the unzipped issue #1 attachments.
+ internal const string DirectoryVariable = "SIMDCODECS_ISSUE1_DIR";
+
+ [Fact]
+ public void EveryCorpusFile_DecodesAsOpenJpegDoes()
+ {
+ var directory = Environment.GetEnvironmentVariable(DirectoryVariable);
+ if (string.IsNullOrWhiteSpace(directory))
+ Assert.Skip($"{DirectoryVariable} is not set; see this class's remarks for where issue #1's files are.");
+
+ OpenJpegOracle.RequireOrSkip();
+
+ var files = Directory.EnumerateFiles(directory, "*", SearchOption.AllDirectories)
+ .Where(f => Path.GetExtension(f).ToLowerInvariant() is ".j2k" or ".j2c" or ".jp2" or ".jpx")
+ .Order(StringComparer.Ordinal)
+ .ToList();
+ files.ShouldNotBeEmpty($"{DirectoryVariable} names {directory}, which holds no JPEG 2000 files.");
+
+ var failures = new List();
+ foreach (var path in files)
+ {
+ var name = Path.GetFileName(path);
+ try
+ {
+ var bytes = File.ReadAllBytes(path);
+ var decoded = Jpeg2000Decoder.Decode(bytes, TestContext.Current.CancellationToken);
+ var expected = OpenJpegOracle.DecodeFile(
+ bytes, Path.GetExtension(path), decoded.Components == 1 ? ".pgm" : ".ppm");
+
+ if (decoded.Width != expected.Width || decoded.Height != expected.Height ||
+ decoded.Components != expected.Components)
+ {
+ failures.Add($"{name}: {decoded.Width}x{decoded.Height}x{decoded.Components}, OpenJPEG " +
+ $"{expected.Width}x{expected.Height}x{expected.Components}");
+ continue;
+ }
+
+ // A reversible codestream must match exactly; one that uses the 9/7 wavelet anywhere
+ // may differ by the rounding of its last step.
+ var tolerance = UsesIrreversibleWavelet(bytes) ? 1 : 0;
+ var worst = 0;
+ for (var i = 0; i < decoded.Samples.Length; i++)
+ worst = Math.Max(worst, Math.Abs(decoded.Samples[i] - expected.Samples[i]));
+ if (worst > tolerance)
+ failures.Add($"{name}: differs from OpenJPEG by up to {worst} (allowed {tolerance})");
+ }
+ catch (Exception e) when (e is InvalidDataException or NotSupportedException)
+ {
+ failures.Add($"{name}: {e.GetType().Name}: {e.Message}");
+ }
+ }
+
+ failures.ShouldBeEmpty();
+ }
+
+ /// Whether any COD or COC in the main header picks the 9/7 wavelet.
+ private static bool UsesIrreversibleWavelet(byte[] bytes)
+ {
+ var data = bytes.AsSpan();
+ if (Jpeg2000Decoder.IsJp2(data))
+ {
+ var layout = Jp2Reader.Read(data);
+ data = data.Slice(layout.CodestreamStart, layout.CodestreamLength);
+ }
+
+ var header = CodestreamReader.Read(data);
+ return header.ComponentCoding.Any(c => c.Transform == WaveletTransform.Irreversible97);
+ }
+}
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jbig2DecoderTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jbig2DecoderTests.cs
index 526bb46..465e019 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Jbig2DecoderTests.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/Jbig2DecoderTests.cs
@@ -183,18 +183,15 @@ public void Decode_UnknownSegmentType_IsSkippedByItsDataLength()
}
///
- /// What is left unimplemented, and it is now one thing: custom Huffman table
- /// segments (§7.4.13). Every region and dictionary type this list used to
- /// name — symbol dictionary, text, refinement, pattern, halftone — decodes,
- /// so a malformed one of those fails as corrupt data instead, which their own
- /// test files cover.
+ /// Custom Huffman table segments (§7.4.13) are decoded now; one too short to hold
+ /// its fixed fields is corrupt data. Jbig2HuffmanTests covers well-formed ones.
///
[Fact]
- public void Decode_CustomHuffmanTableSegment_SaysWhatIsMissing()
+ public void Decode_TruncatedHuffmanTableSegment_IsInvalidData()
{
var stream = Jbig2StreamBuilder.Segment(1, SegmentType.Tables, 1, [0x00, 0x00, 0x00, 0x00]);
- Should.Throw(() => Jbig2Decoder.Decode(stream, 8, 8))
+ Should.Throw(() => Jbig2Decoder.Decode(stream, 8, 8, TestContext.Current.CancellationToken))
.Message.ShouldContain("Huffman table");
}
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jbig2ExtendedOracleTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jbig2ExtendedOracleTests.cs
new file mode 100644
index 0000000..b9a3f8c
--- /dev/null
+++ b/tests/Sdcb.SimdCodecs.Tests/Jbig2ExtendedOracleTests.cs
@@ -0,0 +1,579 @@
+using System.Buffers.Binary;
+using Sdcb.SimdCodecs.Jbig2;
+using Shouldly;
+
+namespace Sdcb.SimdCodecs.Tests;
+
+///
+/// The JBIG2 features added for issue #1, each in a hand-assembled stream checked
+/// against jbig2dec: HENABLESKIP, GRAYMAX above 255, an MMR pattern dictionary,
+/// generic regions of unknown data length (§7.2.7), Huffman-coded symbol
+/// dictionaries and text regions with standard and custom tables, and multi-page
+/// files. The issue's own files are not in the repository (their licence is
+/// unknown); Issue1Jbig2CorpusTests runs them when they are on disk.
+///
+/// The streams' coded bytes come from encoders written here, so a round trip
+/// alone would only prove that encoder and decoder share a reading. jbig2dec is
+/// what makes them checks: every test decodes the same bytes with it too.
+///
+///
+public sealed class Jbig2ExtendedOracleTests
+{
+ private static CancellationToken Ct => TestContext.Current.CancellationToken;
+
+ // ---- halftone ------------------------------------------------------------------
+
+ ///
+ /// HENABLESKIP with a grid that hangs off the region on each side, so cells are
+ /// skipped on the left, top, right and bottom, plus a sheared lattice.
+ ///
+ [Theory]
+ [InlineData(-6 * 256, -5 * 256, 1024, 0)]
+ [InlineData(-2 * 256, 0, 1024, 0)]
+ [InlineData(0, -9 * 256, 1024, 0)]
+ [InlineData(-3 * 256, -3 * 256, 1024, 256)]
+ [InlineData(4 * 256, 2 * 256, 1280, 512)]
+ public void HalftoneWithSkip_AgreesWithJbig2dec(int gridX, int gridY, int vectorX, int vectorY)
+ {
+ Jbig2Oracle.RequireOrSkip();
+
+ const int levels = 4, gridWidth = 9, gridHeight = 7, width = 22, height = 15;
+ var patterns = Jbig2HalftoneTests.Patterns(levels);
+ var grey = Ramp(gridWidth * gridHeight, levels);
+
+ var file = HalftoneFile(width, height, patterns, Jbig2HalftoneTests.PatternDictionarySegment(patterns),
+ Jbig2HalftoneTests.HalftoneRegionSegment(width, height, grey, gridWidth, gridHeight, levels, patterns,
+ gridX, gridY, vectorX, vectorY, enableSkip: true));
+
+ var expected = Stamp(width, height, patterns, grey, gridWidth, gridHeight, gridX, gridY, vectorX, vectorY);
+ ShouldAgree(file, expected);
+ }
+
+ /// GRAYMAX 299: more patterns than a byte can count, nine bitplanes.
+ [Fact]
+ public void GrayMaxAbove255_AgreesWithJbig2dec()
+ {
+ Jbig2Oracle.RequireOrSkip();
+
+ const int levels = 300, gridWidth = 6, gridHeight = 5;
+ var patterns = Jbig2HalftoneTests.Patterns(levels, 18, 17);
+ var grey = new int[gridWidth * gridHeight];
+ for (var i = 0; i < grey.Length; i++) grey[i] = i * 37 % levels;
+
+ var width = gridWidth * 18;
+ var height = gridHeight * 17;
+ var file = HalftoneFile(width, height, patterns, Jbig2HalftoneTests.PatternDictionarySegment(patterns),
+ Jbig2HalftoneTests.HalftoneRegionSegment(width, height, grey, gridWidth, gridHeight, levels, patterns));
+
+ ShouldAgree(file, Stamp(width, height, patterns, grey, gridWidth, gridHeight, 0, 0, 18 * 256, 0));
+ }
+
+ ///
+ /// HDMMR = 1: the collective bitmap is Group 4, and the bytes come from libtiff,
+ /// not from anything in this repository.
+ ///
+ [Fact]
+ public void MmrPatternDictionary_AgreesWithJbig2dec()
+ {
+ Jbig2Oracle.RequireOrSkip();
+
+ const int levels = 8, gridWidth = 6, gridHeight = 5;
+ var patterns = Jbig2HalftoneTests.Patterns(levels);
+ var collective = new Jbig2Bitmap(levels * 4, 4);
+ for (var i = 0; i < levels; i++) collective.Combine(patterns[i], i * 4, 0, CombinationOperator.Replace);
+
+ var dictionary = new List { 0x01, 4, 4 }; // HDMMR = 1, HDPW, HDPH
+ WriteUInt32(dictionary, levels - 1);
+ dictionary.AddRange(Group4Tiff.Encode(collective.Data, collective.Width, collective.Height).Coded);
+
+ var grey = Ramp(gridWidth * gridHeight, levels);
+ var file = HalftoneFile(gridWidth * 4, gridHeight * 4, patterns, [.. dictionary],
+ Jbig2HalftoneTests.HalftoneRegionSegment(gridWidth * 4, gridHeight * 4, grey, gridWidth, gridHeight, levels, patterns));
+
+ ShouldAgree(file, Stamp(gridWidth * 4, gridHeight * 4, patterns, grey, gridWidth, gridHeight, 0, 0, 1024, 0));
+ }
+
+ // ---- unknown data length -------------------------------------------------------
+
+ ///
+ /// An arithmetic generic region with data length 0xFFFFFFFF, ended by FF AC and a
+ /// row count smaller than the declared height (§7.4.6.4), in an embedded stream
+ /// as PDF carries them.
+ ///
+ [Fact]
+ public void UnknownLengthArithmeticRegion_AgreesWithJbig2dec()
+ {
+ Jbig2Oracle.RequireOrSkip();
+
+ const int width = 37, height = 29, rows = 21;
+ var page = Pattern(width, rows);
+ var at = GenericRegionDecoder.NominalAt(0).ToArray();
+ var data = new List(RegionInfo(width, height));
+ data.Add(0x00); // MMR = 0, GBTEMPLATE 0
+ data.AddRange(at.Select(a => (byte)a));
+ // The MQ flush (T.88 E.3.2) already ends on the FF AC marker, which is
+ // exactly the end sequence §7.2.7 asks for; the row count follows it.
+ var coded = Jbig2StreamBuilder.EncodeGenericRegion(page, 0, false, at);
+ coded[^2..].ShouldBe(new byte[] { 0xFF, 0xAC });
+ data.AddRange(coded);
+ WriteUInt32(data, rows);
+
+ var stream = Jbig2StreamBuilder.Stream(
+ Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1, Jbig2StreamBuilder.PageInformation(width, height)),
+ UnknownLength(Jbig2StreamBuilder.Segment(1, SegmentType.ImmediateGenericRegion, 1, [.. data])));
+
+ var expected = new Jbig2Bitmap(width, height);
+ expected.Combine(page, 0, 0, CombinationOperator.Or);
+
+ Jbig2Oracle.Decode(stream, embedded: true).Bits.ShouldBe(expected.Data);
+ Jbig2Decoder.Decode(stream, width, height, Ct).Bits.ToArray().ShouldBe(expected.Data);
+ }
+
+ /// The same with MMR (libtiff's bytes) and the 00 00 end sequence.
+ [Fact]
+ public void UnknownLengthMmrRegion_AgreesWithJbig2dec()
+ {
+ Jbig2Oracle.RequireOrSkip();
+
+ const int width = 64, height = 40;
+ var page = Pattern(width, height);
+ var coded = Group4Tiff.Encode(page.Data, width, height).Coded;
+
+ // 00 00 cannot occur inside T.6 codes, but a trailing zero pad byte would run
+ // into the end sequence and make it ambiguous; libtiff ends on EOFB, so check.
+ coded[^1].ShouldNotBe((byte)0);
+
+ var data = new List(RegionInfo(width, height));
+ data.Add(0x01); // MMR = 1
+ data.AddRange(coded);
+ data.AddRange([0x00, 0x00]);
+ WriteUInt32(data, height);
+
+ var stream = Jbig2StreamBuilder.Stream(
+ Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1, Jbig2StreamBuilder.PageInformation(width, height)),
+ UnknownLength(Jbig2StreamBuilder.Segment(1, SegmentType.ImmediateGenericRegion, 1, [.. data])));
+
+ Jbig2Oracle.Decode(stream, embedded: true).Bits.ShouldBe(page.Data);
+ Jbig2Decoder.Decode(stream, width, height, Ct).Bits.ToArray().ShouldBe(page.Data);
+ }
+
+ [Fact]
+ public void UnknownLength_WithoutEndSequence_IsInvalidData()
+ {
+ var data = new List(RegionInfo(16, 16)) { 0x00, 3, 0xFF, 0xFD, 0xFF, 2, 0xFE, 0xFE, 0xFE, 0x12, 0x34 };
+ var stream = Jbig2StreamBuilder.Stream(
+ Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1, Jbig2StreamBuilder.PageInformation(16, 16)),
+ UnknownLength(Jbig2StreamBuilder.Segment(1, SegmentType.ImmediateGenericRegion, 1, [.. data])));
+
+ Should.Throw(() => Jbig2Decoder.Decode(stream, 16, 16, Ct))
+ .Message.ShouldContain("end sequence");
+ }
+
+ [Fact]
+ public void UnknownLength_InARandomAccessFile_IsInvalidData()
+ {
+ var region = Jbig2StreamBuilder.Split(1, SegmentType.ImmediateGenericRegion, 1, [.. RegionInfo(8, 8), 0x01, 0, 0, 0, 0, 0, 8]);
+ var header = region.Header.ToArray();
+ header.AsSpan(header.Length - 4).Fill(0xFF);
+ var file = Jbig2StreamBuilder.RandomAccessFile(
+ Jbig2StreamBuilder.Split(0, SegmentType.PageInformation, 1, Jbig2StreamBuilder.PageInformation(8, 8)),
+ (header, region.Data));
+
+ Should.Throw(() => Jbig2Decoder.DecodeFile(file, Ct));
+ }
+
+ // ---- Huffman -------------------------------------------------------------------
+
+ public static TheoryData HuffmanVariants => new() { "standard", "custom-dh", "mmr-collective" };
+
+ ///
+ /// A Huffman-coded symbol dictionary (SDHUFF = 1) and a Huffman-coded text region
+ /// (SBHUFF = 1) that stamps its symbols, against jbig2dec: standard tables
+ /// throughout, a custom table segment for SDHUFFDH, and MMR collective bitmaps
+ /// (libtiff's bytes) in place of uncompressed ones.
+ ///
+ [Theory]
+ [MemberData(nameof(HuffmanVariants))]
+ public void HuffmanDictionaryAndTextRegion_AgreeWithJbig2dec(string variant)
+ {
+ Jbig2Oracle.RequireOrSkip();
+
+ var (file, expected) = HuffmanFile(variant);
+ ShouldAgree(file, expected);
+ }
+
+ // ---- pages ---------------------------------------------------------------------
+
+ ///
+ /// A two-page file whose pages differ in size, with a page-0 dictionary-free
+ /// generic region on each: every page decodes to its own raster, and page 1 is
+ /// still what jbig2dec writes first.
+ ///
+ [Fact]
+ public void MultiPageFile_DecodesEveryPage()
+ {
+ var first = Pattern(30, 20);
+ var second = Pattern(41, 13, seed: 5);
+
+ var file = MultiPageFile(first, second);
+
+ Jbig2Decoder.TryReadFileInfo(file, out var w, out var h, out var pages).ShouldBeTrue();
+ (w, h, pages).ShouldBe((30, 20, 2));
+
+ Jbig2Decoder.DecodeFile(file, 1, Ct).Bits.ToArray().ShouldBe(first.Data);
+ var page2 = Jbig2Decoder.DecodeFile(file, 2, Ct);
+ (page2.Width, page2.Height).ShouldBe((41, 13));
+ page2.Bits.ToArray().ShouldBe(second.Data);
+
+ Should.Throw(() => Jbig2Decoder.DecodeFile(file, 3, Ct));
+ Should.Throw(() => Jbig2Decoder.DecodeFile(file, 0, Ct));
+
+ if (Jbig2Oracle.IsAvailable) Jbig2Oracle.Decode(file).Bits.ShouldBe(first.Data);
+ }
+
+ // ---- builders ------------------------------------------------------------------
+
+ private static void ShouldAgree(byte[] file, Jbig2Bitmap expected)
+ {
+ var oracle = Jbig2Oracle.Decode(file);
+ (oracle.Width, oracle.Height).ShouldBe((expected.Width, expected.Height));
+ oracle.Bits.ShouldBe(expected.Data, "jbig2dec");
+ Jbig2Decoder.DecodeFile(file, Ct).Bits.ToArray().ShouldBe(expected.Data, "Sdcb.SimdCodecs.Jbig2");
+ }
+
+ private static int[] Ramp(int count, int levels)
+ {
+ var grey = new int[count];
+ for (var i = 0; i < count; i++) grey[i] = (i * 7 + i / 5) % levels;
+ return grey;
+ }
+
+ private static Jbig2Bitmap Stamp(
+ int width, int height, Jbig2Bitmap[] patterns, int[] grey, int gridWidth, int gridHeight,
+ int gridX, int gridY, int vectorX, int vectorY)
+ {
+ var expected = new Jbig2Bitmap(width, height);
+ for (var m = 0; m < gridHeight; m++)
+ for (var n = 0; n < gridWidth; n++)
+ expected.Combine(patterns[grey[m * gridWidth + n]],
+ (gridX + m * vectorY + n * vectorX) >> 8, (gridY + m * vectorX - n * vectorY) >> 8,
+ CombinationOperator.Or);
+ return expected;
+ }
+
+ private static byte[] HalftoneFile(int width, int height, Jbig2Bitmap[] patterns, byte[] dictionary, byte[] region) =>
+ Jbig2StreamBuilder.SequentialFile(
+ Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1, Jbig2StreamBuilder.PageInformation(width, height)),
+ Jbig2StreamBuilder.Segment(1, SegmentType.PatternDictionary, 1, dictionary),
+ Jbig2StreamBuilder.Segment(2, SegmentType.ImmediateHalftoneRegion, 1, region, referredTo: [1]),
+ Jbig2StreamBuilder.Segment(3, SegmentType.EndOfPage, 1, []));
+
+ /// A deterministic bitmap with runs, edges and isolated pixels.
+ private static Jbig2Bitmap Pattern(int width, int height, int seed = 1)
+ {
+ var bitmap = new Jbig2Bitmap(width, height);
+ for (var y = 0; y < height; y++)
+ for (var x = 0; x < width; x++)
+ bitmap.Data[y * width + x] = (byte)((((x + seed) / 3 + y / 2) % 4 == 0 || (x * 7 + y * 13 + seed) % 23 == 0) ? 1 : 0);
+ return bitmap;
+ }
+
+ private static List RegionInfo(int width, int height)
+ {
+ var body = new List();
+ WriteUInt32(body, (uint)width);
+ WriteUInt32(body, (uint)height);
+ WriteUInt32(body, 0);
+ WriteUInt32(body, 0);
+ body.Add(0); // OR
+ return body;
+ }
+
+ /// Rewrites a built segment's data length field to 0xFFFFFFFF.
+ private static byte[] UnknownLength(byte[] segment)
+ {
+ // Short-form header with no referred-to segments and a one-byte page:
+ // number(4) flags(1) count(1) page(1) length(4).
+ segment.AsSpan(7, 4).Fill(0xFF);
+ return segment;
+ }
+
+ private static byte[] MultiPageFile(Jbig2Bitmap first, Jbig2Bitmap second)
+ {
+ var at = GenericRegionDecoder.NominalAt(0).ToArray();
+ var file = new List(Jbig2Decoder.FileSignature.ToArray()) { 0x01 };
+ WriteUInt32(file, 2);
+ file.AddRange(Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1,
+ Jbig2StreamBuilder.PageInformation(first.Width, first.Height)));
+ file.AddRange(Jbig2StreamBuilder.Segment(1, SegmentType.ImmediateGenericRegion, 1,
+ Jbig2StreamBuilder.GenericRegionSegment(first)));
+ file.AddRange(Jbig2StreamBuilder.Segment(2, SegmentType.EndOfPage, 1, []));
+ file.AddRange(Jbig2StreamBuilder.Segment(3, SegmentType.PageInformation, 2,
+ Jbig2StreamBuilder.PageInformation(second.Width, second.Height)));
+ file.AddRange(Jbig2StreamBuilder.Segment(4, SegmentType.ImmediateGenericRegion, 2,
+ Jbig2StreamBuilder.GenericRegionSegment(second)));
+ file.AddRange(Jbig2StreamBuilder.Segment(5, SegmentType.EndOfPage, 2, []));
+ file.AddRange(Jbig2StreamBuilder.Segment(6, SegmentType.EndOfFile, 0, []));
+ return [.. file];
+ }
+
+ ///
+ /// Builds the Huffman test file: a page, optionally a custom table segment, a
+ /// Huffman symbol dictionary of glyphs in three height classes, and a Huffman text
+ /// region placing them on two strips.
+ ///
+ private static (byte[] File, Jbig2Bitmap Expected) HuffmanFile(string variant)
+ {
+ // Glyphs: heights 5, 7, 7, 9, 9, 9 with varied widths, each a distinct shape.
+ (int W, int H)[] sizes = [(4, 5), (6, 5), (3, 7), (8, 7), (5, 9), (7, 9), (11, 9)];
+ var glyphs = new Jbig2Bitmap[sizes.Length];
+ for (var i = 0; i < sizes.Length; i++)
+ {
+ var (w, h) = sizes[i];
+ glyphs[i] = new Jbig2Bitmap(w, h);
+ for (var y = 0; y < h; y++)
+ for (var x = 0; x < w; x++)
+ glyphs[i].Data[y * w + x] = (byte)((x == 0 || y == h - 1 || (x + y + i) % 3 == 0) ? 1 : 0);
+ }
+
+ var customDh = variant == "custom-dh";
+ var mmr = variant == "mmr-collective";
+
+ // Custom table for SDHUFFDH: HTPS = 2, HTRS = 3, HTLOW = 1, HTHIGH = 17. Lines:
+ // 1..8 (PREFLEN 1, RANGELEN 3), 9..16 (2, 3), lower (3), upper (3).
+ HuffmanTable.Line[] customLines =
+ [
+ new(1, 3, 1), new(2, 3, 9),
+ new(3, 32, 0, HuffmanTable.LineKind.Lower), new(3, 32, 17, HuffmanTable.LineKind.Upper),
+ ];
+ var table = new BitWriter();
+ table.Write(1, 2); table.Write(3, 3);
+ table.Write(2, 2); table.Write(3, 3);
+ table.Write(3, 2); table.Write(3, 2);
+ var tableSegment = new List { (byte)((1 << 1) | (2 << 4)) }; // HTOOB = 0, HTPS - 1 = 1, HTRS - 1 = 2
+ WriteUInt32(tableSegment, 1);
+ WriteUInt32(tableSegment, 17);
+ tableSegment.AddRange(table.ToArray());
+
+ // ---- symbol dictionary (§7.4.2), SDHUFF = 1, SDREFAGG = 0.
+ var dh = customDh ? customLines : HuffmanTable.StandardLines(4);
+ var dw = HuffmanTable.StandardLines(2);
+ var b1 = HuffmanTable.StandardLines(1);
+
+ var sd = new BitWriter();
+ var height = 0;
+ foreach (var group in glyphs.Select((g, i) => (g, i)).GroupBy(t => t.g.Height))
+ {
+ HuffmanEncode(sd, dh, group.Key - height);
+ height = group.Key;
+
+ var width = 0;
+ var members = group.Select(t => t.g).ToArray();
+ foreach (var glyph in members)
+ {
+ HuffmanEncode(sd, dw, glyph.Width - width);
+ width = glyph.Width;
+ }
+ HuffmanEncodeOob(sd, dw);
+
+ var total = members.Sum(g => g.Width);
+ var collective = new Jbig2Bitmap(total, height);
+ var x0 = 0;
+ foreach (var glyph in members) { collective.Combine(glyph, x0, 0, CombinationOperator.Replace); x0 += glyph.Width; }
+
+ if (mmr)
+ {
+ var coded = Group4Tiff.Encode(collective.Data, total, height).Coded;
+ HuffmanEncode(sd, b1, coded.Length);
+ sd.Align();
+ sd.WriteBytes(coded);
+ }
+ else
+ {
+ HuffmanEncode(sd, b1, 0);
+ sd.Align();
+ for (var y = 0; y < height; y++)
+ {
+ for (var x = 0; x < total; x++) sd.Write(collective.Data[y * total + x], 1);
+ sd.Align();
+ }
+ }
+ }
+
+ // Export everything: a zero-length skip run, then all of them.
+ HuffmanEncode(sd, b1, 0);
+ HuffmanEncode(sd, b1, glyphs.Length);
+
+ var dictionary = new List();
+ WriteUInt16(dictionary, (ushort)(0x0001 | (customDh ? 0x000C : 0))); // SDHUFF, SDHUFFDH
+ WriteUInt32(dictionary, (uint)glyphs.Length);
+ WriteUInt32(dictionary, (uint)glyphs.Length);
+ dictionary.AddRange(sd.ToArray());
+
+ // ---- text region (§7.4.3), SBHUFF = 1, REFCORNER = TOPLEFT, one strip line each.
+ const int pageWidth = 70, pageHeight = 40;
+ (int Id, int S, int T)[][] strips =
+ [
+ [(0, 2, 3), (3, 8, 3), (1, 17, 3), (6, 24, 3), (2, 36, 3)],
+ [(5, 5, 18), (4, 13, 18), (0, 20, 18), (6, 26, 18), (3, 40, 18), (1, 49, 18)],
+ ];
+ var instances = strips.Sum(s => s.Length);
+
+ var codeLength = 3; // seven symbols: every code length 3, so symbol i is code i
+ var tr = new BitWriter();
+ for (var r = 0; r < 35; r++) tr.Write(r == codeLength ? 1u : 0u, 4); // run code lengths
+ for (var i = 0; i < glyphs.Length; i++) tr.Write(0, 1); // RUNCODE3 = "0", once per symbol
+ tr.Align();
+
+ var fs = HuffmanTable.StandardLines(6);
+ var ds = HuffmanTable.StandardLines(8);
+ var dt = HuffmanTable.StandardLines(11);
+
+ HuffmanEncode(tr, dt, 1); // initial STRIPT = -1
+ var stript = -1;
+ var firsts = 0;
+ foreach (var strip in strips)
+ {
+ HuffmanEncode(tr, dt, strip[0].T - stript);
+ stript = strip[0].T;
+ HuffmanEncode(tr, fs, strip[0].S - firsts);
+ firsts = strip[0].S;
+
+ var curs = strip[0].S;
+ for (var k = 0; k < strip.Length; k++)
+ {
+ tr.Write((uint)strip[k].Id, codeLength);
+ curs += glyphs[strip[k].Id].Width - 1;
+ if (k + 1 < strip.Length)
+ {
+ HuffmanEncode(tr, ds, strip[k + 1].S - curs);
+ curs = strip[k + 1].S;
+ }
+ else
+ {
+ HuffmanEncodeOob(tr, ds);
+ }
+ }
+ }
+
+ var text = new List(RegionInfo(pageWidth, pageHeight));
+ WriteUInt16(text, 0x0001 | (1 << 4)); // SBHUFF, REFCORNER = TOPLEFT
+ WriteUInt16(text, 0x0000); // FS B.6, DS B.8, DT B.11
+ WriteUInt32(text, (uint)instances);
+ text.AddRange(tr.ToArray());
+
+ var segments = new List
+ {
+ Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1,
+ Jbig2StreamBuilder.PageInformation(pageWidth, pageHeight)),
+ };
+ if (customDh) segments.Add(Jbig2StreamBuilder.Segment(1, SegmentType.Tables, 1, [.. tableSegment]));
+ segments.Add(Jbig2StreamBuilder.Segment(2, SegmentType.SymbolDictionary, 1, [.. dictionary],
+ referredTo: customDh ? [1] : null));
+ segments.Add(Jbig2StreamBuilder.Segment(3, SegmentType.ImmediateLosslessTextRegion, 1, [.. text], referredTo: [2]));
+ segments.Add(Jbig2StreamBuilder.Segment(4, SegmentType.EndOfPage, 1, []));
+
+ var expected = new Jbig2Bitmap(pageWidth, pageHeight);
+ foreach (var strip in strips)
+ foreach (var (id, s, t) in strip)
+ expected.Combine(glyphs[id], s, t, CombinationOperator.Or);
+
+ return (Jbig2StreamBuilder.SequentialFile([.. segments]), expected);
+ }
+
+ ///
+ /// Writes with a table given as lines, assigning the
+ /// prefix codes by T.88 B.3 here rather than through the decoder's table.
+ ///
+ private static void HuffmanEncode(BitWriter writer, HuffmanTable.Line[] lines, int value)
+ {
+ var codes = Codes(lines);
+ for (var i = 0; i < lines.Length; i++)
+ {
+ var line = lines[i];
+ if (line.PrefixLength == 0) continue;
+ switch (line.Kind)
+ {
+ case HuffmanTable.LineKind.Normal when value >= line.RangeLow && value < line.RangeLow + (1L << line.RangeLength):
+ writer.Write(codes[i], line.PrefixLength);
+ writer.Write((uint)(value - line.RangeLow), line.RangeLength);
+ return;
+ case HuffmanTable.LineKind.Lower when value <= line.RangeLow:
+ writer.Write(codes[i], line.PrefixLength);
+ writer.Write((uint)(line.RangeLow - value), 32);
+ return;
+ case HuffmanTable.LineKind.Upper when value >= line.RangeLow:
+ writer.Write(codes[i], line.PrefixLength);
+ writer.Write((uint)(value - line.RangeLow), 32);
+ return;
+ }
+ }
+
+ throw new ArgumentOutOfRangeException(nameof(value), value, "Not representable in this table.");
+ }
+
+ private static void HuffmanEncodeOob(BitWriter writer, HuffmanTable.Line[] lines)
+ {
+ var codes = Codes(lines);
+ var i = Array.FindIndex(lines, l => l.Kind == HuffmanTable.LineKind.OutOfBand);
+ writer.Write(codes[i], lines[i].PrefixLength);
+ }
+
+ private static uint[] Codes(HuffmanTable.Line[] lines)
+ {
+ var max = lines.Max(l => l.PrefixLength);
+ var codes = new uint[lines.Length];
+ uint first = 0;
+ var previous = 0;
+ for (var length = 1; length <= max; length++)
+ {
+ first = (uint)((first + previous) << 1);
+ var current = first;
+ previous = 0;
+ for (var i = 0; i < lines.Length; i++)
+ if (lines[i].PrefixLength == length) { codes[i] = current++; previous++; }
+ }
+ return codes;
+ }
+
+ private sealed class BitWriter
+ {
+ private readonly List _bytes = [];
+ private int _bits;
+
+ public void Write(uint value, int count)
+ {
+ for (var i = count - 1; i >= 0; i--)
+ {
+ if (_bits % 8 == 0) _bytes.Add(0);
+ if (((value >> i) & 1) != 0) _bytes[^1] |= (byte)(0x80 >> (_bits % 8));
+ _bits++;
+ }
+ }
+
+ public void Align() => _bits = (_bits + 7) & ~7;
+
+ public void WriteBytes(byte[] data)
+ {
+ Align();
+ _bytes.AddRange(data);
+ _bits += data.Length * 8;
+ }
+
+ public byte[] ToArray() => [.. _bytes];
+ }
+
+ private static void WriteUInt16(List target, ushort value)
+ {
+ Span buffer = stackalloc byte[2];
+ BinaryPrimitives.WriteUInt16BigEndian(buffer, value);
+ target.AddRange(buffer);
+ }
+
+ private static void WriteUInt32(List target, uint value)
+ {
+ Span buffer = stackalloc byte[4];
+ BinaryPrimitives.WriteUInt32BigEndian(buffer, value);
+ target.AddRange(buffer);
+ }
+}
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jbig2FacadeTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jbig2FacadeTests.cs
index 39d22ae..28219af 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Jbig2FacadeTests.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/Jbig2FacadeTests.cs
@@ -144,9 +144,9 @@ public void TryDecode_OnAFileNeedingAnUnimplementedFeature_ReturnsFalse()
{
// The facade contract is "false for undecodable", not "throws". Every
// region type decodes now, so the case has to be a genuinely refused
- // feature: a Huffman-coded symbol dictionary.
+ // feature: a symbol dictionary that imports arithmetic contexts.
var dictionary = Jbig2SymbolBuilder.SymbolDictionarySegment(Jbig2TextRegionTests.Alphabet());
- dictionary[1] |= 0x01; // SDHUFF
+ dictionary[0] |= 0x01; // bitmap coding context used (bit 8 of the flags)
var file = Jbig2StreamBuilder.SequentialFile(
Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1,
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jbig2HalftoneTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jbig2HalftoneTests.cs
index 1e77d89..a9233cd 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Jbig2HalftoneTests.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/Jbig2HalftoneTests.cs
@@ -62,36 +62,24 @@ public void HalftoneRegion_WithNoPatternDictionary_Fails()
.Message.ShouldContain("pattern dictionary");
}
+ ///
+ /// GRAYMAX is a four-byte field; one that asks for a collective bitmap past the
+ /// per-bitmap ceiling is refused before anything is allocated.
+ ///
[Fact]
- public void HalftoneRegion_WithSkipEnabled_IsRefusedByName()
- {
- var patterns = Patterns(4);
- var segment = HalftoneRegionSegment(32, 32, new int[4], 2, 2, 4, patterns);
- segment[17] |= 0x08; // HENABLESKIP, in the flags byte after the region info
-
- var stream = Jbig2StreamBuilder.Stream(
- Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1,
- Jbig2StreamBuilder.PageInformation(32, 32)),
- Jbig2StreamBuilder.Segment(1, SegmentType.PatternDictionary, 1, PatternDictionarySegment(patterns)),
- Jbig2StreamBuilder.Segment(2, SegmentType.ImmediateHalftoneRegion, 1, segment, referredTo: [1]));
-
- Should.Throw(() => Jbig2Decoder.Decode(stream, 32, 32))
- .Message.ShouldContain("HENABLESKIP");
- }
-
- [Fact]
- public void MmrCodedPatternDictionary_IsRefusedByName()
+ public void PatternDictionary_ImplausibleGrayMax_IsRefusedBeforeAllocating()
{
var segment = PatternDictionarySegment(Patterns(4));
- segment[0] |= 0x01; // HDMMR
+ segment[3] = segment[4] = segment[5] = segment[6] = 0xFF; // GRAYMAX = 0xFFFFFFFF
var stream = Jbig2StreamBuilder.Stream(
Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1,
Jbig2StreamBuilder.PageInformation(32, 32)),
Jbig2StreamBuilder.Segment(1, SegmentType.PatternDictionary, 1, segment));
- Should.Throw(() => Jbig2Decoder.Decode(stream, 32, 32))
- .Message.ShouldContain("MMR");
+ Should.Throw(() => Jbig2Decoder.Decode(stream, 32, 32,
+ TestContext.Current.CancellationToken))
+ .Message.ShouldContain("GRAYMAX");
}
// ---- stream construction -----------------------------------------------------
@@ -199,27 +187,46 @@ internal static byte[] PatternDictionarySegment(Jbig2Bitmap[] patterns, int temp
}
/// A halftone region segment (T.88 §7.4.5) carrying the grey grid as Gray-coded bitplanes.
+ ///
+ /// HENABLESKIP: cells whose pattern lies wholly outside the region are left out of
+ /// every bitplane (T.88 §6.6.5.1). The skip test is written out here a second time,
+ /// independently of the decoder's, so the two only agree with jbig2dec if both are right.
+ ///
internal static byte[] HalftoneRegionSegment(
int width, int height, int[] grey, int gridWidth, int gridHeight, int levels,
Jbig2Bitmap[] patterns, int gridX = 0, int gridY = 0, int vectorX = 0, int vectorY = 0,
- int template = 0)
+ int template = 0, bool enableSkip = false)
{
if (vectorX == 0) vectorX = patterns[0].Width * 256;
+ bool[]? skip = null;
+ if (enableSkip)
+ {
+ skip = new bool[gridWidth * gridHeight];
+ for (var m = 0; m < gridHeight; m++)
+ for (var n = 0; n < gridWidth; n++)
+ {
+ var x = (gridX + m * vectorY + n * vectorX) >> 8;
+ var y = (gridY + m * vectorX - n * vectorY) >> 8;
+ skip[m * gridWidth + n] = x + patterns[0].Width <= 0 || x >= width
+ || y + patterns[0].Height <= 0 || y >= height;
+ }
+ }
+
var body = new List();
WriteUInt32(body, (uint)width);
WriteUInt32(body, (uint)height);
WriteUInt32(body, 0);
WriteUInt32(body, 0);
body.Add((byte)CombinationOperator.Or); // external combination
- body.Add((byte)(template << 1)); // HMMR=0, HENABLESKIP=0, HCOMBOP=OR, HDEFPIXEL=0
+ body.Add((byte)((template << 1) | (enableSkip ? 0x08 : 0))); // HMMR=0, HCOMBOP=OR, HDEFPIXEL=0
WriteUInt32(body, (uint)gridWidth);
WriteUInt32(body, (uint)gridHeight);
WriteUInt32(body, (uint)gridX);
WriteUInt32(body, (uint)gridY);
WriteUInt16(body, (ushort)vectorX);
WriteUInt16(body, (ushort)vectorY);
- body.AddRange(EncodeGrayscale(grey, gridWidth, gridHeight, levels, template));
+ body.AddRange(EncodeGrayscale(grey, gridWidth, gridHeight, levels, template, skip));
return [.. body];
}
@@ -229,7 +236,7 @@ internal static byte[] HalftoneRegionSegment(
/// one arithmetic stream with one shared context array, exactly as the decoder
/// reads them back.
///
- private static byte[] EncodeGrayscale(int[] values, int width, int height, int levels, int template)
+ private static byte[] EncodeGrayscale(int[] values, int width, int height, int levels, int template, bool[]? skip = null)
{
var planes = 0;
while (1 << planes < levels) planes++;
@@ -249,11 +256,21 @@ private static byte[] EncodeGrayscale(int[] values, int width, int height, int l
plane.Data[i] = previous is null ? raw : (byte)(raw ^ previous[i]);
}
+ // A skipped cell is coded nowhere and decodes as 0 in every plane, so
+ // its plane bit has to be 0 here too for the contexts of its
+ // neighbours to match the decoder's.
+ if (skip is not null)
+ for (var i = 0; i < values.Length; i++)
+ if (skip[i]) plane.Data[i] = 0;
+
for (var y = 0; y < height; y++)
for (var x = 0; x < width; x++)
+ {
+ if (skip is not null && skip[y * width + x]) continue;
encoder.Encode(contexts,
GenericRegionDecoder.Context(plane, x, y, template, at),
plane.Data[y * width + x]);
+ }
// The next plane down is coded against this one *after* decoding, so
// the running reference is the reconstructed bit, not the coded one.
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jbig2HuffmanTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jbig2HuffmanTests.cs
new file mode 100644
index 0000000..e3d31c7
--- /dev/null
+++ b/tests/Sdcb.SimdCodecs.Tests/Jbig2HuffmanTests.cs
@@ -0,0 +1,118 @@
+using Sdcb.SimdCodecs.Jbig2;
+using Shouldly;
+
+namespace Sdcb.SimdCodecs.Tests;
+
+///
+/// The Huffman layer of T.88 Annex B: the standard tables, canonical code
+/// assignment, custom table segments, and their resource bounds. Whether the
+/// tables are the spec's is settled against jbig2dec in
+/// Jbig2ExtendedOracleTests and on the issue #1 corpus.
+///
+public sealed class Jbig2HuffmanTests
+{
+ ///
+ /// Every standard table is a prefix code whose Kraft sum is exactly 1, except B.7,
+ /// whose lines leave 1/16 of the code space unused. A misremembered line length
+ /// would almost always break this.
+ ///
+ [Theory]
+ [InlineData(1)] [InlineData(2)] [InlineData(3)] [InlineData(4)] [InlineData(5)]
+ [InlineData(6)] [InlineData(7)] [InlineData(8)] [InlineData(9)] [InlineData(10)]
+ [InlineData(11)] [InlineData(12)] [InlineData(13)] [InlineData(14)] [InlineData(15)]
+ public void StandardTable_IsAPrefixCode(int number)
+ {
+ var sum = HuffmanTable.StandardLines(number)
+ .Where(l => l.PrefixLength > 0)
+ .Sum(l => Math.ScaleB(1.0, -l.PrefixLength));
+
+ sum.ShouldBe(number == 7 ? 15.0 / 16 : 1.0);
+ _ = HuffmanTable.Get(number);
+ }
+
+ /// B.3's codes as the table prints them, decoded bit by bit.
+ [Theory]
+ [InlineData("0", 0)]
+ [InlineData("10", 1)]
+ [InlineData("110", 2)]
+ [InlineData("1110101", 8)] // 1110 + 3 bits: 3 + 5
+ [InlineData("11110000011", 14)] // 11110 + 6 bits: 11 + 3
+ [InlineData("111110", HuffmanTable.OutOfBand)]
+ [InlineData("1111110" + "00000000000000000000000000000101", 80)] // upper: 75 + 5
+ [InlineData("11111110" + "11111111", -1)] // -256 + 255
+ [InlineData("11111111" + "00000000000000000000000000000011", -260)] // lower: -257 - 3
+ public void TableB3_DecodesItsPrintedCodes(string bits, int value)
+ {
+ var data = Pack(bits);
+ var reader = new Jbig2BitReader(data);
+ HuffmanTable.Get(3).Decode(ref reader).ShouldBe(value);
+ }
+
+ ///
+ /// A custom table segment (§7.4.13): HTPS = 2, HTRS = 3, values 1 to 16 in two
+ /// lines, then the lower and upper range lines.
+ ///
+ [Fact]
+ public void TableSegment_BuildsItsLines()
+ {
+ // flags: HTOOB = 0, HTPS - 1 = 1, HTRS - 1 = 2; HTLOW = 1; HTHIGH = 17;
+ // lines (PREFLEN, RANGELEN) = (1, 3), (2, 3); lower PREFLEN 3; upper PREFLEN 3.
+ byte[] segment = [0x22, 0, 0, 0, 1, 0, 0, 0, 17, .. Pack("01011" + "10011" + "11" + "11")];
+ var table = HuffmanTable.ReadSegment(segment);
+
+ Decode(table, "0" + "000").ShouldBe(1);
+ Decode(table, "0" + "111").ShouldBe(8);
+ Decode(table, "10" + "101").ShouldBe(14);
+ Decode(table, "110" + new string('0', 31) + "1").ShouldBe(-1); // lower: HTLOW - 1 - 1
+ Decode(table, "111" + new string('0', 30) + "10").ShouldBe(19); // upper: HTHIGH + 2
+ }
+
+ [Fact]
+ public void TableSegment_WithLowNotBelowHigh_IsInvalidData()
+ {
+ byte[] segment = [0x22, 0, 0, 0, 9, 0, 0, 0, 9, 0xFF, 0xFF];
+ Should.Throw(() => HuffmanTable.ReadSegment(segment));
+ }
+
+ ///
+ /// HTLOW to HTHIGH spanning the whole int range with RANGELEN 0 would be four
+ /// billion lines; each one costs bits of the segment, so the reader runs out of
+ /// data long before it runs out of memory.
+ ///
+ [Fact]
+ public void TableSegment_DeclaringBillionsOfLines_FailsOnItsData()
+ {
+ byte[] segment = [0x00, 0x80, 0, 0, 0, 0x7F, 0xFF, 0xFF, 0xFF, .. new byte[64]];
+ Should.Throw(() => HuffmanTable.ReadSegment(segment));
+ }
+
+ [Fact]
+ public void Lengths_OverSubscribingTheCodeSpace_AreInvalidData()
+ {
+ Should.Throw(() => HuffmanTable.FromCodeLengths([1, 1, 1]));
+ }
+
+ [Fact]
+ public void Reader_PastTheEnd_IsInvalidData()
+ {
+ Should.Throw(() =>
+ {
+ var reader = new Jbig2BitReader([0xFF]);
+ reader.ReadBits(9);
+ });
+ }
+
+ private static int Decode(HuffmanTable table, string bits)
+ {
+ var reader = new Jbig2BitReader(Pack(bits));
+ return table.Decode(ref reader);
+ }
+
+ private static byte[] Pack(string bits)
+ {
+ var bytes = new byte[(bits.Length + 7) / 8];
+ for (var i = 0; i < bits.Length; i++)
+ if (bits[i] == '1') bytes[i >> 3] |= (byte)(0x80 >> (i & 7));
+ return bytes;
+ }
+}
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jbig2Oracle.cs b/tests/Sdcb.SimdCodecs.Tests/Jbig2Oracle.cs
index 974313d..8ec3adc 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Jbig2Oracle.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/Jbig2Oracle.cs
@@ -130,6 +130,31 @@ private static bool TryVersion(string exe, string[] prefix, out string output)
/// file header) over a standalone .jb2 file.
///
public static Raster Decode(byte[] stream, bool embedded = false)
+ {
+ var pbm = Run(stream, embedded);
+ var position = 0;
+ return ParsePbm(pbm, ref position);
+ }
+
+ ///
+ /// Decodes every page of : jbig2dec writes a
+ /// multi-page file as consecutive PBM images in one output.
+ ///
+ public static IReadOnlyList DecodePages(byte[] stream, bool embedded = false)
+ {
+ var pbm = Run(stream, embedded);
+ var pages = new List();
+ var position = 0;
+ while (position < pbm.Length && ReadToken(pbm, ref position) is { Length: > 0 } magic)
+ {
+ position -= magic.Length;
+ pages.Add(ParsePbm(pbm, ref position));
+ }
+
+ return pages;
+ }
+
+ private static byte[] Run(byte[] stream, bool embedded)
{
var (exe, viaWsl, _) = Resolved.Value;
if (exe is null) throw new InvalidOperationException("jbig2dec oracle is not available.");
@@ -184,7 +209,7 @@ public static Raster Decode(byte[] stream, bool embedded = false)
throw new InvalidOperationException(
$"jbig2dec produced no output (exit {process.ExitCode}).\n{stdout}\n{stderr}");
- return ParsePbm(File.ReadAllBytes(output));
+ return File.ReadAllBytes(output);
}
finally
{
@@ -209,10 +234,8 @@ private static string ToWslPath(string path)
/// and a set bit means black — so the unpacked result is already in T.88
/// polarity and needs no inversion.
///
- private static Raster ParsePbm(byte[] pbm)
+ private static Raster ParsePbm(byte[] pbm, ref int position)
{
- var position = 0;
-
if (ReadToken(pbm, ref position) != "P4")
throw new InvalidDataException("jbig2dec did not produce a binary PBM (P4).");
@@ -234,6 +257,7 @@ private static Raster ParsePbm(byte[] pbm)
bits[y * width + x] = (byte)((pbm[row + (x >> 3)] >> (7 - (x & 7))) & 1);
}
+ position += stride * height;
return new Raster(width, height, bits);
}
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jbig2RefinementTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jbig2RefinementTests.cs
index cc0b1d7..6476065 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Jbig2RefinementTests.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/Jbig2RefinementTests.cs
@@ -29,9 +29,8 @@ public static TheoryData Cases()
var data = new TheoryData();
foreach (var edit in (string[])["identical", "speckle", "grow", "shrink", "shift", "inverted"])
foreach (var template in (int[])[0, 1])
- // TPGRON is refused rather than decoded — see
- // RefinementRegionDecoder's remarks and the refusal test below.
- data.Add(edit, template, false);
+ foreach (var tpgron in (bool[])[false, true])
+ data.Add(edit, template, tpgron);
return data;
}
@@ -135,47 +134,27 @@ public void RefinementRegion_RefinesThePageUnderItsOwnRectangle()
}
///
- /// TPGRON is refused. §6.3 is otherwise confirmed against jbig2dec, but with
- /// this flag the two decoders disagree in ways that measurement narrowed and
- /// did not resolve — see RefinementRegionDecoder's remarks. "Mostly agrees"
- /// is not a decoder a caller can use, so it throws.
+ /// TPGRON survives the segment layer: a refinement segment with the flag set
+ /// decodes to its target. The pixel-level agreement with jbig2dec is
+ /// Jbig2RefinementOracleTests.
///
[Theory]
[InlineData(0)]
[InlineData(1)]
- public void Tpgron_IsRefusedByName(int template)
+ public void Tpgron_InARefinementSegment_Decodes(int template)
{
- var (reference, target) = Build("identical", 16, 8);
- var contexts = new byte[1 << RefinementRegionDecoder.ContextBits(template)];
-
- var exception = Should.Throw(() =>
- {
- var mq = new MqDecoder(new byte[16]);
- RefinementRegionDecoder.Decode(
- ref mq, contexts, target.Width, target.Height, template,
- reference, 0, 0, typicalPrediction: true, RefinementRegionDecoder.NominalAt,
- Jbig2PixelBudget.Unmetered());
- });
-
- exception.Message.ShouldContain("TPGRON");
- }
-
- /// And the refusal has to survive the segment layer, not just the inner call.
- [Fact]
- public void Tpgron_InARefinementSegment_IsRefused()
- {
- var (reference, target) = Build("identical", 16, 8);
+ var (reference, target) = Build("speckle", 40, 24);
var stream = Jbig2StreamBuilder.Stream(
Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1,
- Jbig2StreamBuilder.PageInformation(16, 8)),
+ Jbig2StreamBuilder.PageInformation(40, 24)),
Jbig2StreamBuilder.Segment(1, SegmentType.ImmediateGenericRegion, 1,
Jbig2StreamBuilder.GenericRegionSegment(reference)),
Jbig2StreamBuilder.Segment(2, SegmentType.ImmediateRefinementRegion, 1,
- Jbig2StreamBuilder.RefinementRegionSegment(target, reference, typicalPrediction: true)));
+ Jbig2StreamBuilder.RefinementRegionSegment(
+ target, reference, op: CombinationOperator.Replace, template: template, typicalPrediction: true)));
- Should.Throw(() => Jbig2Decoder.Decode(stream, 16, 8))
- .Message.ShouldContain("TPGRON");
+ Jbig2Decoder.Decode(stream, 40, 24, TestContext.Current.CancellationToken).Bits.ToArray().ShouldBe(target.Data);
}
[Fact]
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jbig2SegmentTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jbig2SegmentTests.cs
index 2cff7e6..1363d0c 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Jbig2SegmentTests.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/Jbig2SegmentTests.cs
@@ -171,11 +171,12 @@ public void ReadHeader_PageAssociationFlag_SelectsTheFourByteForm()
}
[Fact]
- public void ReadHeader_UnknownDataLength_IsRejectedExplicitly()
+ public void ReadHeader_UnknownDataLength_OnAnImmediateGenericRegion_IsLeftToTheStreamReader()
{
// §7.2.7 allows 0xFFFFFFFF on an immediate generic region, whose data is
- // then terminated by a search sequence instead. Not implemented, and it
- // must say so rather than treat the value as a length.
+ // then terminated by an end sequence instead. The header reader reports
+ // it rather than treating the value as a length; Jbig2ExtendedOracleTests
+ // covers the resolution.
byte[] header =
[
0x00, 0x00, 0x00, 0x01,
@@ -186,7 +187,25 @@ public void ReadHeader_UnknownDataLength_IsRejectedExplicitly()
];
var position = 0;
- Should.Throw(() => Jbig2Segment.ReadHeader(header, ref position))
+ var segment = Jbig2Segment.ReadHeader(header, ref position);
+ segment.LengthUnknown.ShouldBeTrue();
+ segment.DataStart.ShouldBe(header.Length);
+ }
+
+ [Fact]
+ public void ReadHeader_UnknownDataLength_OnAnyOtherType_IsInvalidData()
+ {
+ byte[] header =
+ [
+ 0x00, 0x00, 0x00, 0x01,
+ 0x06, // immediate text region
+ 0x00,
+ 0x01,
+ 0xFF, 0xFF, 0xFF, 0xFF,
+ ];
+
+ var position = 0;
+ Should.Throw(() => Jbig2Segment.ReadHeader(header, ref position))
.Message.ShouldContain("unknown data length");
}
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jbig2TextRegionTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jbig2TextRegionTests.cs
index 947b409..c0868d8 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Jbig2TextRegionTests.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/Jbig2TextRegionTests.cs
@@ -114,38 +114,6 @@ public void TextRegion_WithNoSymbolDictionary_Fails()
.Message.ShouldContain("symbol dictionary");
}
- [Fact]
- public void HuffmanCodedTextRegion_IsRefusedByName()
- {
- var segment = Jbig2SymbolBuilder.TextRegionSegment(16, 16, Alphabet(), [new(0, 0, 0)]);
- segment[18] |= 0x01; // SBHUFF, low byte of the flags at offset 17..18
-
- var stream = Jbig2StreamBuilder.Stream(
- Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1,
- Jbig2StreamBuilder.PageInformation(16, 16)),
- Jbig2StreamBuilder.Segment(1, SegmentType.SymbolDictionary, 1,
- Jbig2SymbolBuilder.SymbolDictionarySegment(Alphabet())),
- Jbig2StreamBuilder.Segment(2, SegmentType.ImmediateTextRegion, 1, segment, referredTo: [1]));
-
- Should.Throw(() => Jbig2Decoder.Decode(stream, 16, 16))
- .Message.ShouldContain("SBHUFF");
- }
-
- [Fact]
- public void HuffmanCodedSymbolDictionary_IsRefusedByName()
- {
- var segment = Jbig2SymbolBuilder.SymbolDictionarySegment(Alphabet());
- segment[1] |= 0x01; // SDHUFF, low byte of the 16-bit flags
-
- var stream = Jbig2StreamBuilder.Stream(
- Jbig2StreamBuilder.Segment(0, SegmentType.PageInformation, 1,
- Jbig2StreamBuilder.PageInformation(16, 16)),
- Jbig2StreamBuilder.Segment(1, SegmentType.SymbolDictionary, 1, segment));
-
- Should.Throw(() => Jbig2Decoder.Decode(stream, 16, 16))
- .Message.ShouldContain("SDHUFF");
- }
-
///
/// A dictionary that imports from another — the mechanism PDF's
/// /JBIG2Globals is built on, and the one place symbol numbering spans
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jpeg2000Jp2Tests.cs b/tests/Sdcb.SimdCodecs.Tests/Jpeg2000Jp2Tests.cs
index a265013..8d170db 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Jpeg2000Jp2Tests.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/Jpeg2000Jp2Tests.cs
@@ -94,21 +94,52 @@ public void OnlyTheFirstColourSpecification_Counts()
}
///
- /// The three boxes that change what the components mean are refused by name. Skipped, each
- /// would hand back samples whose meaning had silently changed: palette indices as if they were
- /// colour, channels out of order, an alpha channel drawn as colour.
+ /// The channel definition box is refused by name: skipped, it would hand back an alpha channel
+ /// drawn as colour, or channels out of order. (The palette and component mapping boxes are
+ /// decoded; .)
+ ///
+ [Fact]
+ public void ChannelDefinition_IsRefused()
+ {
+ var wrapped = Jp2(
+ Header(ImageHeader(), EnumeratedColour(16), Box("cdef", [0, 0, 0, 0])),
+ Box("jp2c", Codestream("rgb-struct64")));
+
+ Should.Throw(() => Jpeg2000Decoder.Decode(wrapped)).Message.ShouldContain("channel definition");
+ }
+
+ ///
+ /// Palette and component mapping boxes that do not describe a palette are malformed: an empty
+ /// palette, a palette with no mapping, a mapping into a column the palette lacks, a palette
+ /// shorter than its entries, an undefined mapping type, a mapping naming a missing component.
///
[Theory]
- [InlineData("pclr", "palette")]
- [InlineData("cmap", "component mapping")]
- [InlineData("cdef", "channel definition")]
- public void BoxesThatChangeTheMeaningOfComponents_AreRefused(string type, string named)
+ [InlineData("empty palette")]
+ [InlineData("palette without mapping")]
+ [InlineData("missing column")]
+ [InlineData("short palette")]
+ [InlineData("bad mapping type")]
+ [InlineData("missing component")]
+ [InlineData("too many entries")]
+ public void MalformedPaletteOrMapping_IsInvalid(string what)
{
+ byte[] palette = [0, 2, 1, 7, 10, 20];
+ byte[][] boxes = what switch
+ {
+ "empty palette" => [Box("pclr", [0, 0, 0, 0]), Box("cmap", [0, 0, 1, 0])],
+ "palette without mapping" => [Box("pclr", palette)],
+ "missing column" => [Box("pclr", palette), Box("cmap", [0, 0, 1, 1])],
+ "short palette" => [Box("pclr", [0, 200, 1, 7, 1, 2, 3]), Box("cmap", [0, 0, 1, 0])],
+ "bad mapping type" => [Box("pclr", palette), Box("cmap", [0, 0, 2, 0])],
+ "too many entries" => [Box("pclr", [0xFF, 0xFF, 1, 7]), Box("cmap", [0, 0, 1, 0])],
+ _ => [Box("pclr", palette), Box("cmap", [0, 9, 1, 0])],
+ };
+
var wrapped = Jp2(
- Header(ImageHeader(), EnumeratedColour(16), Box(type, [0, 0, 0, 0])),
+ Header([ImageHeader(), EnumeratedColour(16), .. boxes]),
Box("jp2c", Codestream("rgb-struct64")));
- Should.Throw(() => Jpeg2000Decoder.Decode(wrapped)).Message.ShouldContain(named);
+ Should.Throw(() => Jpeg2000Decoder.Decode(wrapped));
}
///
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jpeg2000ResourceLimitTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jpeg2000ResourceLimitTests.cs
index 60656ba..0350792 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Jpeg2000ResourceLimitTests.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/Jpeg2000ResourceLimitTests.cs
@@ -169,9 +169,51 @@ public void AnUnmeteredBudget_NeverRunsOut()
[InlineData("cblk4-struct64")]
[InlineData("odd37x23")]
[InlineData("flat64")]
- public void SingleByteMutations_NeverEscapeTheContract(string name)
+ public void SingleByteMutations_NeverEscapeTheContract(string name) => Mutate(Fixture(name));
+
+ ///
+ /// The same sweep over a codestream using every tier-2 feature at once: tiles of odd sizes, an
+ /// odd origin, tile-parts, precincts, a position-driven progression, SOP and EPH markers and
+ /// segmentation symbols. Made live by opj_compress, so it needs the oracle.
+ ///
+ [Fact]
+ public void SingleByteMutations_OfATiledPrecinctCodestream_NeverEscapeTheContract()
+ {
+ OpenJpegOracle.RequireOrSkip();
+
+ Mutate(OpenJpegOracle.Compress(Jpeg2000TilingAndProgressionTests.Picture(53, 41, 1),
+ "-t", "23,19", "-d", "1,2", "-T", "1,1", "-n", "3", "-c", "[16,16]", "-p", "RPCL", "-TP", "R",
+ "-SOP", "-EPH", "-M", "34", "-r", "10,1"));
+ }
+
+ ///
+ /// A header declaring thousands of tiles in front of the data for one. Every tile needs a
+ /// tile-part, so it is refused as soon as the tile-parts are counted, before any geometry is
+ /// built for the tiles that are missing.
+ ///
+ [Fact]
+ public void ManyDeclaredTiles_WithTheDataForOne_AreRefused()
+ {
+ var bytes = Fixture("dwt5-struct64");
+ BinaryPrimitives.WriteUInt32BigEndian(bytes.AsSpan(24), 1);
+ BinaryPrimitives.WriteUInt32BigEndian(bytes.AsSpan(28), 1);
+
+ Should.Throw(() => Jpeg2000Decoder.Decode(bytes)).Message.ShouldContain("tile-parts");
+ }
+
+ /// More tiles than a 16-bit tile index can name is not a JPEG 2000 codestream.
+ [Fact]
+ public void MoreTilesThanATileIndexCanName_AreRefused()
+ {
+ var bytes = WithDeclaredSize("dwt5-struct64", 1000, 1000);
+ BinaryPrimitives.WriteUInt32BigEndian(bytes.AsSpan(24), 1);
+ BinaryPrimitives.WriteUInt32BigEndian(bytes.AsSpan(28), 1);
+
+ Should.Throw(() => Jpeg2000Decoder.Decode(bytes)).Message.ShouldContain("65,535");
+ }
+
+ private static void Mutate(byte[] original)
{
- var original = Fixture(name);
// A fixed sweep rather than a random one: the same bytes are tried on
// every run, so a failure is reproducible from the test name alone.
@@ -187,7 +229,7 @@ public void SingleByteMutations_NeverEscapeTheContract(string name)
var image = Jpeg2000Decoder.Decode(mutated, cancellationToken: TestContext.Current.CancellationToken);
// If it decoded, the result must at least be self-consistent.
- image.Samples.Length.ShouldBe(image.Width * image.Height);
+ image.Samples.Length.ShouldBe(image.Width * image.Height * image.Components);
}
catch (InvalidDataException)
{
diff --git a/tests/Sdcb.SimdCodecs.Tests/Jpeg2000TilingAndProgressionTests.cs b/tests/Sdcb.SimdCodecs.Tests/Jpeg2000TilingAndProgressionTests.cs
new file mode 100644
index 0000000..8835851
--- /dev/null
+++ b/tests/Sdcb.SimdCodecs.Tests/Jpeg2000TilingAndProgressionTests.cs
@@ -0,0 +1,209 @@
+using System.Buffers.Binary;
+using System.Text;
+using Sdcb.SimdCodecs.Jpeg2000;
+using Shouldly;
+
+namespace Sdcb.SimdCodecs.Tests;
+
+///
+/// Tier-2 beyond one tile and one precinct, made live by opj_compress and checked against
+/// opj_decompress: several tiles (odd sizes, an odd image and tile origin), tiles split into
+/// tile-parts, custom precincts, all five progression orders, SOP and EPH markers, the code-block
+/// styles that are decoded, and the JP2 palette. The reversible codestreams must decode to OpenJPEG's
+/// samples exactly; the irreversible ones to within one, as in .
+///
+/// Issue #1's corpus () is what showed these were missing;
+/// these are the committed, licence-clean version of the same coverage.
+///
+///
+public class Jpeg2000TilingAndProgressionTests
+{
+ ///
+ /// A deterministic test picture: a gradient with structure and some noise, so every subband has
+ /// something to code and no two tiles look alike.
+ ///
+ internal static byte[] Picture(int width, int height, int components)
+ {
+ var header = Encoding.ASCII.GetBytes($"P{(components == 3 ? 6 : 5)}\n{width} {height}\n255\n");
+ var pixels = new byte[width * height * components];
+ var seed = 12345u;
+ for (var y = 0; y < height; y++)
+ {
+ for (var x = 0; x < width; x++)
+ {
+ for (var c = 0; c < components; c++)
+ {
+ seed = seed * 1664525 + 1013904223;
+ var smooth = (x * (3 + c) + y * (5 - c)) % 256;
+ var stripe = ((x / 7 + y / 5 + c) & 1) * 40;
+ pixels[(y * width + x) * components + c] = (byte)Math.Clamp(smooth / 2 + stripe + (int)(seed >> 28), 0, 255);
+ }
+ }
+ }
+
+ return [.. header, .. pixels];
+ }
+
+ public static TheoryData Reversible => new()
+ {
+ { "tiles, odd sizes", "-t 37,29 -n 3" },
+ { "tiles, odd image and tile origin", "-t 37,29 -d 3,5 -T 1,2 -n 3" },
+ { "tiles, odd origin, one-sample resolutions", "-t 9,7 -d 1,1 -n 3" },
+ { "RLCP, layers", "-p RLCP -r 20,5,1 -t 64,64" },
+ { "RPCL, precincts", "-p RPCL -c [32,32],[16,16] -r 10,1" },
+ { "PCRL, precincts", "-p PCRL -c [64,64],[32,32],[16,16] -r 8,2" },
+ { "CPRL, precincts", "-p CPRL -c [32,32] -n 4 -r 6,1" },
+ { "LRCP, precincts and tiles", "-c [16,16] -t 48,40 -n 3 -r 5,1" },
+ { "precincts smaller than code-blocks", "-c [16,16] -b 16,16 -n 4" },
+ { "SOP and EPH", "-SOP -EPH -t 64,64 -r 10,1" },
+ { "tile-parts by resolution", "-TP R -t 64,64 -p RPCL -r 10,1" },
+ { "tile-parts by component", "-TP C -t 48,48 -p CPRL" },
+ { "tile-parts by layer", "-TP L -t 64,64 -r 20,5,1" },
+ { "segmentation symbols", "-M 32 -t 64,64" },
+ { "context reset", "-M 2" },
+ { "predictable termination", "-M 16" },
+ { "reset, termination and segmentation together", "-M 50 -p RPCL -c [32,32]" },
+ };
+
+ [Theory]
+ [MemberData(nameof(Reversible))]
+ public void GreyCodestream_DecodesExactlyAsOpenJpegDoes(string what, string options)
+ {
+ OpenJpegOracle.RequireOrSkip();
+ _ = what;
+
+ var codestream = OpenJpegOracle.Compress(Picture(157, 123, 1), options.Split(' '));
+ AssertMatches(codestream, ".pgm", tolerance: 0);
+ }
+
+ [Theory]
+ [MemberData(nameof(Reversible))]
+ public void RgbCodestream_DecodesExactlyAsOpenJpegDoes(string what, string options)
+ {
+ OpenJpegOracle.RequireOrSkip();
+ _ = what;
+
+ var codestream = OpenJpegOracle.Compress(Picture(101, 89, 3), options.Split(' '));
+ AssertMatches(codestream, ".ppm", tolerance: 0);
+ }
+
+ public static TheoryData Irreversible => new()
+ {
+ { "tiles, odd origin", "-I -t 37,29 -d 3,5 -T 1,2 -n 3 -r 20" },
+ { "tiles, odd origin, one-sample resolutions", "-I -t 9,7 -d 1,1 -n 3" },
+ { "RPCL, precincts, SOP, EPH, segmentation", "-I -p RPCL -c [32,32],[16,16] -SOP -EPH -M 32 -t 64,64 -r 40,10" },
+ { "PCRL, tile-parts", "-I -p PCRL -TP R -t 48,48 -r 30" },
+ { "CPRL, layers", "-I -p CPRL -c [64,64] -r 50,20,5" },
+ };
+
+ [Theory]
+ [MemberData(nameof(Irreversible))]
+ public void IrreversibleCodestream_DecodesWithinOneOfOpenJpeg(string what, string options)
+ {
+ OpenJpegOracle.RequireOrSkip();
+ _ = what;
+
+ AssertMatches(OpenJpegOracle.Compress(Picture(157, 123, 1), options.Split(' ')), ".pgm", tolerance: 1);
+ AssertMatches(OpenJpegOracle.Compress(Picture(101, 89, 3), options.Split(' ')), ".ppm", tolerance: 1);
+ }
+
+ ///
+ /// A reduced decode of a tiled image: every tile is reduced on its own, and the reduced tiles
+ /// must still meet edge to edge as OpenJPEG's -r puts them.
+ ///
+ [Theory]
+ [InlineData(1)]
+ [InlineData(2)]
+ [InlineData(3)]
+ public void TiledCodestream_ReducesAsOpenJpegDoes(int reduce)
+ {
+ OpenJpegOracle.RequireOrSkip();
+
+ var codestream = OpenJpegOracle.Compress(Picture(157, 123, 1), "-t", "37,29", "-d", "3,5", "-T", "1,2", "-n", "4", "-p", "RPCL");
+ var decoded = Jpeg2000Decoder.Decode(codestream, reduce, TestContext.Current.CancellationToken);
+ var expected = OpenJpegOracle.Decode(codestream, ".pgm", "-r", reduce.ToString());
+
+ decoded.Width.ShouldBe(expected.Width);
+ decoded.Height.ShouldBe(expected.Height);
+ decoded.Samples.ShouldBe(expected.Samples);
+ }
+
+ ///
+ /// A JP2 palette (pclr) with its component mapping (cmap): one component of indices turned into
+ /// three of colour, checked against OpenJPEG decoding the same file.
+ ///
+ [Fact]
+ public void PalettedJp2_DecodesAsOpenJpegDoes()
+ {
+ OpenJpegOracle.RequireOrSkip();
+
+ // Indices 0..199 into a 200-entry palette.
+ const int width = 61, height = 47, entries = 200;
+ var indices = new byte[width * height];
+ for (var i = 0; i < indices.Length; i++) indices[i] = (byte)((i * 7 + i / width * 3) % entries);
+ byte[] pgm = [.. Encoding.ASCII.GetBytes($"P5\n{width} {height}\n255\n"), .. indices];
+ var codestream = OpenJpegOracle.Compress(pgm, "-t", "32,32");
+
+ var palette = new List { (byte)(entries >> 8), (byte)(entries & 0xFF), 3, 7, 7, 7 };
+ for (var e = 0; e < entries; e++) palette.AddRange([(byte)(e * 5 % 256), (byte)(255 - e), (byte)(e * e % 256)]);
+
+ var file = Jp2(
+ Box("jp2h", [
+ .. Box("ihdr", [.. BigEndian32(height), .. BigEndian32(width), 0, 1, 7, 7, 0, 0]),
+ .. Box("colr", [1, 0, 0, .. BigEndian32(16)]),
+ .. Box("pclr", [.. palette]),
+ .. Box("cmap", [0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 2]),
+ ]),
+ Box("jp2c", codestream));
+
+ var decoded = Jpeg2000Decoder.Decode(file, TestContext.Current.CancellationToken);
+ var expected = OpenJpegOracle.DecodeFile(file, ".jp2", ".ppm");
+
+ decoded.Components.ShouldBe(3);
+ decoded.BitDepth.ShouldBe(8);
+ decoded.Width.ShouldBe(expected.Width);
+ decoded.Samples.ShouldBe(expected.Samples);
+
+ var info = Jpeg2000Decoder.ReadInfo(file);
+ info.Components.ShouldBe(3);
+ info.BitDepth.ShouldBe(8);
+ }
+
+ private static void AssertMatches(byte[] codestream, string extension, int tolerance)
+ {
+ var decoded = Jpeg2000Decoder.Decode(codestream, TestContext.Current.CancellationToken);
+ var expected = OpenJpegOracle.Decode(codestream, extension);
+
+ decoded.Width.ShouldBe(expected.Width);
+ decoded.Height.ShouldBe(expected.Height);
+ decoded.Components.ShouldBe(expected.Components);
+
+ if (tolerance == 0)
+ {
+ decoded.Samples.ShouldBe(expected.Samples);
+ return;
+ }
+
+ var worst = 0;
+ for (var i = 0; i < decoded.Samples.Length; i++)
+ worst = Math.Max(worst, Math.Abs(decoded.Samples[i] - expected.Samples[i]));
+ worst.ShouldBeLessThanOrEqualTo(tolerance);
+ }
+
+ internal static byte[] Jp2(params byte[][] boxes)
+ {
+ byte[] signature = [0x00, 0x00, 0x00, 0x0C, 0x6A, 0x50, 0x20, 0x20, 0x0D, 0x0A, 0x87, 0x0A];
+ var fileType = Box("ftyp", [.. "jp2 "u8, 0, 0, 0, 0, .. "jp2 "u8]);
+ return [.. signature, .. fileType, .. boxes.SelectMany(b => b)];
+ }
+
+ internal static byte[] Box(string type, byte[] body) =>
+ [.. BigEndian32(8 + body.Length), .. Encoding.ASCII.GetBytes(type), .. body];
+
+ internal static byte[] BigEndian32(int value)
+ {
+ var bytes = new byte[4];
+ BinaryPrimitives.WriteInt32BigEndian(bytes, value);
+ return bytes;
+ }
+}
diff --git a/tests/Sdcb.SimdCodecs.Tests/OpenJpegOracle.cs b/tests/Sdcb.SimdCodecs.Tests/OpenJpegOracle.cs
index 6eb7523..73a2a89 100644
--- a/tests/Sdcb.SimdCodecs.Tests/OpenJpegOracle.cs
+++ b/tests/Sdcb.SimdCodecs.Tests/OpenJpegOracle.cs
@@ -91,6 +91,75 @@ public static Pnm.Image Decode(byte[] codestream, string extension = ".pgm", par
}
}
+ ///
+ /// for a file whose format opj_decompress
+ /// must be told by its name: is .jp2 for a JP2 file.
+ ///
+ public static Pnm.Image DecodeFile(byte[] file, string inputExtension, string outputExtension)
+ {
+ var tools = Resolved.Value;
+ if (tools.Decompress is null) throw new InvalidOperationException("OpenJPEG oracle is not available.");
+
+ var scratch = Path.Combine(AppContext.BaseDirectory, "jpeg2000-oracle-tmp");
+ Directory.CreateDirectory(scratch);
+ var stem = Path.Combine(scratch, Guid.NewGuid().ToString("n"));
+ var input = stem + inputExtension;
+ var output = stem + outputExtension;
+
+ try
+ {
+ File.WriteAllBytes(input, file);
+ var (exitCode, log) = Run(tools.Decompress, ["-i", input, "-o", output], tools.LibraryPath);
+ if (!File.Exists(output))
+ throw new InvalidOperationException(
+ $"opj_decompress produced no output (exit {exitCode}).{Environment.NewLine}{log}");
+
+ return Pnm.Read(output);
+ }
+ finally
+ {
+ TryDelete(input);
+ TryDelete(output);
+ }
+ }
+
+ ///
+ /// Encodes a binary PNM with opj_compress and returns the raw codestream it wrote, so a
+ /// test can make the exact coding options it needs (tiles, precincts, progression orders,
+ /// markers) rather than depend on committed bytes.
+ ///
+ /// A P5 or P6 file.
+ /// opj_compress options, after -i and -o.
+ public static byte[] Compress(byte[] pnm, params string[] args)
+ {
+ var tools = Resolved.Value;
+ if (tools.Compress is null) throw new InvalidOperationException("OpenJPEG oracle is not available.");
+
+ var scratch = Path.Combine(AppContext.BaseDirectory, "jpeg2000-oracle-tmp");
+ Directory.CreateDirectory(scratch);
+ var stem = Path.Combine(scratch, Guid.NewGuid().ToString("n"));
+ var input = stem + (pnm[1] == (byte)'6' ? ".ppm" : ".pgm");
+ var output = stem + ".j2k";
+
+ try
+ {
+ File.WriteAllBytes(input, pnm);
+ var all = new List { "-i", input, "-o", output };
+ all.AddRange(args);
+ var (exitCode, log) = Run(tools.Compress, all, tools.LibraryPath);
+ if (!File.Exists(output))
+ throw new InvalidOperationException(
+ $"opj_compress produced no output (exit {exitCode}).{Environment.NewLine}{log}");
+
+ return File.ReadAllBytes(output);
+ }
+ finally
+ {
+ TryDelete(input);
+ TryDelete(output);
+ }
+ }
+
private static void TryDelete(string path)
{
try { if (File.Exists(path)) File.Delete(path); }
diff --git a/tests/Sdcb.SimdCodecs.Tests/Oracle/jbig2/README.md b/tests/Sdcb.SimdCodecs.Tests/Oracle/jbig2/README.md
index 711c197..732ab98 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Oracle/jbig2/README.md
+++ b/tests/Sdcb.SimdCodecs.Tests/Oracle/jbig2/README.md
@@ -10,6 +10,11 @@ bottom.
| **us → jbig2dec** | `jbig2dec` (Artifex, AGPL) | `Jbig2Oracle` + `Jbig2OracleTests` | Yes, else the tests skip |
| **libtiff → us** (MMR) | ImageMagick / libtiff, via Magick.NET | `Group4Tiff` + `Jbig2MmrOracleTests` | **No** — Magick.NET is already a package reference |
+**Optional local corpus.** The 114 `.jb2` files attached to issue #1 (unknown source and
+licence, so never committed or fetched by a test) are checked page by page against jbig2dec by
+`Issue1Jbig2CorpusTests` when `SIMDCODECS_ISSUE1_DIR` points at their extracted directory; the
+test's remarks give the URLs and the expected result. Run it after any change to the decoder.
+
They cover different things, which is the point of having all three.
- jbig2enc only ever emits **GBTEMPLATE 0 with nominal AT pixels** (asserted by
diff --git a/tests/Sdcb.SimdCodecs.Tests/Oracle/jpeg2000/README.md b/tests/Sdcb.SimdCodecs.Tests/Oracle/jpeg2000/README.md
index 0ce60b2..2c159d7 100644
--- a/tests/Sdcb.SimdCodecs.Tests/Oracle/jpeg2000/README.md
+++ b/tests/Sdcb.SimdCodecs.Tests/Oracle/jpeg2000/README.md
@@ -118,3 +118,18 @@ because their expected output is not their source. Their tests decode them with
Sources are regenerated by `gen-sources.py` from closed-form functions of
`(x, y)` and one fixed LCG — no library PRNG, no floating point — so a
regeneration on another machine or Python version reproduces the same bytes.
+
+## Made at test time: tiles, precincts, progressions, markers, palettes
+
+`Jpeg2000TilingAndProgressionTests` commits no codestream at all. It writes a picture, has
+`opj_compress` code it with the option under test (`-t`, `-d`/`-T` for an odd origin, `-TP`,
+`-c`, `-p RLCP|RPCL|PCRL|CPRL`, `-SOP -EPH`, `-M 2|16|32`, `-I`), and compares with
+`opj_decompress` on the same bytes. Two `opj_compress` 2.5.4 shapes are kept out because
+OpenJPEG cannot round-trip them itself: a tile one sample wide (its own decode is not lossless
+there) and `-TP L` on 48x48 tiles of the 157x123 picture (it cannot decode what it wrote).
+
+## Issue #1's corpus
+
+The 16 JPEG 2000 files attached to issue #1 have no known licence, so they are not committed or
+fetched by a test. `Issue1Jpeg2000CorpusTests` runs them when `SIMDCODECS_ISSUE1_DIR` names the
+unzipped attachments (the URLs are in that class's remarks); run it after changing the decoder.