Skip to content

sbe: import a char in a non-UTF-8 encoding as bytes - #176

Merged
trendvidia merged 2 commits into
mainfrom
sbe/char-encoding-173
Nov 3, 2026
Merged

trendvidia merged 2 commits into
mainfrom
sbe/char-encoding-173

Conversation

@bfrancojr

Copy link
Copy Markdown
Collaborator

Convert.xmlToProto now reads a char type's characterEncoding. A char in an encoding that isn't UTF-8 compatible becomes bytes, not string.

Before, every char became a string. So a schema declaring ISO-8859-1 imported cleanly, and the first byte outside UTF-8 then failed the whole message.

Fixes #173

What changes

  • A char becomes a string only when its encoding is absent (the FIX SBE default, US-ASCII) or names US-ASCII or UTF-8. Names are matched against the IANA names and aliases, ignoring case and punctuation. This is draft -01 § Character Encoding (SBE: declare a character encoding for text fields protowire#494).
  • Any other encoding becomes bytes with the same (sbe.length), plus a comment line that says why. This also applies inside a composite.
  • The list of names and the comment wording follow protowire-go#248. That PR is still open, so the two ports will match once it merges.

Evidence

Classification: reversible by a revert. It changes generated schema output for non-UTF-8 chars only, as the draft requires. Measured.

The only overlap with #171, which also touches sbe, is the CHANGELOG.

@bfrancojr

Copy link
Copy Markdown
Collaborator Author

CI passed on a577c01, the PR head read back from the remote: Build + Test, Conformance (Go reference), Analyze and CodeQL.

Convert.xmlToProto mapped every char to a string and ignored
characterEncoding. A decoder refuses a string that is not UTF-8, so a
char declared ISO-8859-1 imported cleanly and its first byte past ASCII
failed the whole message. The draft now maps a char to string only for
an absent, US-ASCII or UTF-8 encoding, and to bytes of the same length
otherwise (protowire#494). The names and the comment line follow
protowire-go#248.
@trendvidia
trendvidia force-pushed the sbe/char-encoding-173 branch from a577c01 to 3980686 Compare November 3, 2026 08:58
@bfrancojr

Copy link
Copy Markdown
Collaborator Author

I rebased onto main 35d1341. The only conflict was the CHANGELOG, and both entries are kept. The new head is 3980686, read back from the remote. ./gradlew build at the pinned plugin: 735 tests, 0 failures, 0 skipped.

@trendvidia
trendvidia merged commit 0666965 into main Nov 3, 2026
4 checks passed
@trendvidia
trendvidia deleted the sbe/char-encoding-173 branch November 3, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sbe: import a char in a non-UTF-8 encoding as bytes (protowire#494)

2 participants