Skip to content

sbe: keep fixed-length byte arrays through XML import and export - #184

Merged
trendvidia merged 1 commit into
mainfrom
sbe/fixed-arrays-179
Nov 3, 2026
Merged

trendvidia merged 1 commit into
mainfrom
sbe/fixed-arrays-179

Conversation

@bfrancojr

Copy link
Copy Markdown
Collaborator

Fixes #179.

The SBE XML converter lost fixed-length arrays that are not text. This ports protowire-go#251, following the rule in draft -01 § Fixed-Length Arrays.

Import (Convert.xmlToProto)

  • A uint8 or int8 array of length N > 1 becomes bytes with (sbe.length) = N. Before, it became one narrowed uint32, so a 16-byte field was read as 1 byte and every later field in the block moved.
  • An array of a wider primitive is refused with IllegalArgumentException, naming the type, the field and the message. For example: sbe: message "Record": field "uuid": type "Uuid" is an array of 2 uint64, which no proto field represents. Go uses the same text.

Export (Convert.protoToXml)

  • A bytes field with (sbe.length) = N is written as primitiveType="uint8" length="N", declared once per length as bytesN. Before, it was a char array, which imports back as a string. A string stays a char array.
  • A bytes field of length 1 comes back as uint32 [(sbe.encoding) = "uint8"]: the same byte in the block, but not bytes. A test pins this.

The wire bytes do not change. Only the converter's output does.

Tests (FixedArrayTest, using Go's schema and cases)

  • The importer's output for the schema is checked in as arrays/arr.proto and compiled. A test checks it equals what the importer writes now.
  • A codec test checks that the block layout is 45 bytes, that the uuid is at offset 12 to 28, and that the tail is in the last 4 bytes.
  • XML → proto → XML → proto gives the same proto text, except the comment naming the Latin-1 encoding. Same descriptors means the same values encode to the same bytes.
  • Two cases check that a wide array is refused, plus one exact-message check.
  • One test pins the length-1 case.

Evidence

  • ./gradlew :sbe:test: 41 tests, 0 failures. FixedArrayTest ran 9.
  • With Convert.java reverted to main, 7 of the 9 fail. The two that still pass test the codec against the checked-in proto, not the converter.
  • ./gradlew build with the plugin fetched at the pin (-Dorg.gradle.project.pxfJavaMeta.repo=/nonexistent, as CI does): BUILD SUCCESSFUL.

xmlToProto ignored the length of a non-char array, so a uint8 array of 16
imported as one uint32 narrowed to uint8 and every later field in the
block moved. A uint8 or int8 array now imports as bytes with the same
(sbe.length), and an array of a wider primitive is refused, naming the
type, field and message.

protoToXml wrote a bytes field as a char array, which imports back as a
string. It now writes a uint8 array, declared once per length as bytesN.
The wire bytes do not change.

A bytes field of length 1 exports as length="1", which FIX SBE reads as
a single uint8, so it comes back as a narrowed uint32; pinned by a test.

Follows protowire-go#251; the schema and cases are its tests.
@trendvidia
trendvidia merged commit c685ff0 into main Nov 3, 2026
4 checks passed
@trendvidia
trendvidia deleted the sbe/fixed-arrays-179 branch November 3, 2026 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sbe: XML import/export loses fixed-length arrays that are not text

2 participants