Repository navigation
sbe: import a char in a non-UTF-8 encoding as bytes - #176
Merged
Merged
Conversation
Collaborator
Author
|
CI passed on a577c01, the PR head read back from the remote: Build + Test, Conformance (Go reference), Analyze and CodeQL. |
Convert.xmlToProto mapped every char to a string and ignored characterEncoding. A decoder refuses a string that is not UTF-8, so a char declared ISO-8859-1 imported cleanly and its first byte past ASCII failed the whole message. The draft now maps a char to string only for an absent, US-ASCII or UTF-8 encoding, and to bytes of the same length otherwise (protowire#494). The names and the comment line follow protowire-go#248.
trendvidia
force-pushed
the
sbe/char-encoding-173
branch
from
November 3, 2026 08:58
a577c01 to
3980686
Compare
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Convert.xmlToProtonow reads achartype'scharacterEncoding. Acharin an encoding that isn't UTF-8 compatible becomesbytes, notstring.Before, every
charbecame astring. So a schema declaringISO-8859-1imported cleanly, and the first byte outside UTF-8 then failed the whole message.Fixes #173
What changes
charbecomes astringonly when its encoding is absent (the FIX SBE default, US-ASCII) or names US-ASCII or UTF-8. Names are matched against the IANA names and aliases, ignoring case and punctuation. This is draft -01 § Character Encoding (SBE: declare a character encoding for text fields protowire#494).byteswith the same(sbe.length), plus a comment line that says why. This also applies inside a composite.Evidence
./gradlew buildat a577c01: 673 tests, 0 failures, 5 skipped (pb: null and zero big numbers in Pb.marshal differ from protowire-go, and a null list element throws #159's, enabled by pb: write null and zero big numbers as protowire-go does, and a BigFloat normalised #170).CharacterEncodingTest(18 tests) uses the schema of Go'sTestXMLToProtoCharacterEncoding.ASCII,US-ASCIIandutf_8givestring;ISO-8859-1, andwindows-1252inside a composite, givebytes.sbe/src/test/proto/charenc/enc.proto, so the build compiles it. A test asserts the file is the importer's output byte for byte.café(63 61 66 E9) round-trips throughCodecin thebytesfield.Classification: reversible by a revert. It changes generated schema output for non-UTF-8 chars only, as the draft requires. Measured.
The only overlap with #171, which also touches
sbe, is the CHANGELOG.