Skip to content

test(j2k-native): check whole codestreams for marker values in tile data - #320

Merged
jcwal1516 merged 1 commit into
frames-sg:mainfrom
mshunshin:test/codeblock-termination-markers
Nov 3, 2026
Merged

jcwal1516 merged 1 commit into
frames-sg:mainfrom
mshunshin:test/codeblock-termination-markers

Conversation

@mshunshin

Copy link
Copy Markdown
Contributor

Why

#167 drops the classic MQ encoder's terminal 0xFF, as T.800 B.10.7 requires ("The sequence of bytes actually included for any given code-block must not end in a 0xFF"). Its tests check each code-block's bytes: finish_omits_pending_terminal_ff, short_sequences_never_end_in_ff_and_round_trip and classic_code_block_contributions_never_end_in_ff.

Nothing checks what the rule protects on a whole codestream. Code-block data and packet headers follow one another directly, so a trailing 0xFF pairs with the next byte, and a byte-aligned pair in 0xFF90–0xFFFF is a marker value (D.4, note). A decoder resynchronising on SOP/EPH, or looking for the EOC after a tile-part with Psot 0, can stop at one.

Change

One integration test file, crates/j2k-native/tests/codeblock_termination.rs, and no library changes. It encodes through the public encode and finds each tile-part's data from its SOT's Psot and its SOD.

  • tile_data_holds_no_marker_values: no byte-aligned 0xFF90–0xFFFF pair and no final 0xFF in any tile-part's data. It covers 16×9, 64×64 and 256×256 grey images, reversible and irreversible, classic and HTJ2K. The reversible codestreams must also decode back to their samples.
  • packet_bodies_never_end_in_0xff: with SOP and EPH on, no packet body ends in 0xFF, which would otherwise pair with the next packet's SOP.

The samples are noise over a gradient, so code-blocks have enough coding passes that most MQ terminations would end in 0xFF without the fix.

Results

Both tests pass on main at 968dbb4. With the self.data.pop() in ArithmeticEncoder::finish_checked removed, both fail:

16x9 reversible=true ht=false: byte-aligned 0xFFCF in tile data
reversible=true: packet 2's body ends in 0xFF

No changelog entry, since the change is test-only (like #177 and #318).

Run locally (aarch64 macOS):

  • cargo fmt --all --check
  • the strict Clippy gate on this test target, with xtask's pedantic and nursery lints and baseline allows
  • cargo xtask repo-lint
  • cargo test --release -p j2k-native -p j2k-codec-math -p j2k: 1,661 passed, 0 failed

I wrote these tests with the same encoder fix on my fork; #167 landed first, so only the tests are left to offer.

AI-Tooling: Claude Opus 5.5 noreply@anthropic.com

T.800 B.10.7: "The sequence of bytes actually included for any given
code-block must not end in a 0xFF." Code-block data and packet headers
follow one another directly, so a trailing 0xFF pairs with the next byte,
and a pair in 0xFF90..=0xFFFF is a marker value (D.4, note). frames-sg#167 drops
the classic MQ encoder's terminal 0xFF, and its tests check each
code-block's bytes. These check what the rule protects, on whole encoded
codestreams:

- tile_data_holds_no_marker_values: no byte-aligned 0xFF90..=0xFFFF pair
  and no final 0xFF in any tile-part's data, for 16x9, 64x64 and 256x256
  images, reversible and irreversible, classic and HTJ2K. The reversible
  codestreams decode back to their samples.
- packet_bodies_never_end_in_0xff: with SOP and EPH, no packet body ends
  in 0xFF, so the next packet's SOP stays a marker.

Both fail with the terminal-0xFF drop in ArithmeticEncoder::finish_checked
removed (a byte-aligned 0xFFCF in the 16x9 reversible classic tile data,
and a packet body ending in 0xFF) and pass on main.

AI-Tooling: Claude Opus 5.5 <noreply@anthropic.com>
@jcwal1516

Copy link
Copy Markdown
Member

Thanks for catching this! I ran both new tests on my current checkout and they passed.

LGTM, I'll merge once CI is green.

@jcwal1516
jcwal1516 merged commit 5f50b5f into frames-sg:main Nov 3, 2026
30 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants