Skip to content

sbe: keep fixed-length byte arrays through XML import and export #184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
trendvidia merged 1 commit into from
Nov 3, 2026
Merged

sbe: keep fixed-length byte arrays through XML import and export #184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ format changes.

### Changed

- **`Convert.protoToXml` writes a `bytes` field as a `uint8` array** (#179; draft `-01` § Fixed-Length Arrays). A `bytes` field with `(sbe.length) = N` was written as a `char` array, which `Convert.xmlToProto` reads back as a `string`, so a `bytes` field did not survive a round trip. It is now `primitiveType="uint8" length="N"`, declared once per length as `bytesN` beside the `strN` types. The wire bytes are the same. A `bytes` field of length 1 is written as `length="1"`, which FIX SBE reads as a single `uint8`, so it reads back as `uint32` with `(sbe.encoding) = "uint8"`: the same byte in the block, not `bytes`. This follows protowire-go#251.
- **`Convert.xmlToProto` refuses an array of a primitive wider than a byte** (#179). This narrows what imports. A type such as `primitiveType="int32" length="4"` used to import as one `int32`, which misplaced every later field in the block. No proto field holds such an array, so the import now throws `IllegalArgumentException` naming the type, the field and the message.
- **The lite tier prints nothing for a plain field whose wire value is zero** (#180). `LiteWireReader` used to print `f_int32 = 0` for `2800`, where protowire-go and the full tier print nothing: a plain proto3 field has no set state, so a zero on the wire means the same as no field. A field in `PxfMeta.implicitPresenceFields()` whose last value is its zero value is now left out. A `-0.0`, an `optional` field and a map entry's key and value keep their zeros. A companion from an older plugin has an empty table and prints as before.

- **Both tiers write PXF text to the same PB bytes as protowire-go** (#156, #157, #158; the spec's write gate, protowire#511). Each change below used to make a valid document write different bytes from Go, or accept one Go refuses:
Expand All @@ -54,6 +56,7 @@ format changes.

### Fixed

- **`Convert.xmlToProto` keeps the length of a `uint8` or `int8` array** (#179). `<type primitiveType="uint8" length="16"/>` imported as one `uint32` with `(sbe.encoding) = "uint8"`, so a 16-byte field was read as 1 byte and every later field in the block was misplaced. It now imports as `bytes` with `(sbe.length) = 16`, the same 16 bytes on the wire.
- **The SBE decoder rejects a string field that is not UTF-8** (#162; `docs/HARDENING.md` § UTF-8). Both tiers turned the bytes of an `(sbe.length)` `string` into a `String` with `new String(bytes, UTF_8)`, which puts U+FFFD in place of bad bytes and says nothing, so `41 42 FF FE` decoded as `"AB\uFFFD\uFFFD"`. `SbeWireCodec` and `View.stringValue` now decode strictly and throw `IllegalArgumentException` (`sbe: invalid UTF-8 in string field <name>`). A `bytes` field takes any bytes, as before. `check-decode-android` decodes `adversarial.v1.SbeText`, so the lite tier runs the spec's `sbe/utf8-*` rows.

- **An SBE emitter cuts an overlong string on a character boundary** (#166; draft -01 § SBE Annotations, protowire#495). Both tiers cut an `(sbe.length)` value at the byte limit, which can split a multi-byte character, and every strict decoder, this port's now included, then rejects the whole message. `ABCDEFGé` in a string field of length 8 was written `41 42 43 44 45 46 47 C3`. A string is now cut at the last whole character at or before the limit and NUL-padded: `41 42 43 44 45 46 47 00`. A `bytes` field is still cut at the limit. Only bytes that strict decoders refuse change.
Expand Down
94 changes: 74 additions & 20 deletions sbe/src/main/java/org/protowire/sbe/Convert.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,13 @@ static void writeProtoComposite(StringBuilder b, XmlSchema.XmlComposite c) {
b.append("message ").append(c.name).append(" {\n");
int n = 1;
for (XmlSchema.XmlType t : c.types) {
writeTypedField(b, " ", resolveTypeToProto(t), camelToSnake(t.name), n);
String[] mt;
try {
mt = resolveTypeToProto(t);
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("sbe: composite \"" + c.name + "\": " + e.getMessage(), e);
}
writeTypedField(b, " ", mt, camelToSnake(t.name), n);
n++;
}
for (XmlSchema.XmlRef r : c.refs) {
Expand All @@ -86,8 +92,12 @@ static void writeProtoMessage(StringBuilder b, XmlSchema.XmlMessage m,
Map<String, XmlSchema.XmlEnum> enumMap) {
b.append("message ").append(m.name).append(" {\n");
b.append(" option (sbe.template_id) = ").append(m.id).append(";\n");
for (XmlSchema.XmlField f : m.fields) writeProtoField(b, f, typeMap, compMap, enumMap, " ");
for (XmlSchema.XmlGroup g : m.groups) writeProtoGroup(b, g, typeMap, compMap, enumMap, " ");
try {
for (XmlSchema.XmlField f : m.fields) writeProtoField(b, f, typeMap, compMap, enumMap, " ");
for (XmlSchema.XmlGroup g : m.groups) writeProtoGroup(b, g, typeMap, compMap, enumMap, " ");
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("sbe: message \"" + m.name + "\": " + e.getMessage(), e);
}
b.append("}\n\n");
}

Expand All @@ -103,7 +113,13 @@ static void writeProtoField(StringBuilder b, XmlSchema.XmlField f,
}
XmlSchema.XmlType t = typeMap.get(f.type);
if (t != null) {
writeTypedField(b, indent, resolveTypeToProto(t), name, f.id);
String[] mt;
try {
mt = resolveTypeToProto(t);
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("field \"" + f.name + "\": " + e.getMessage(), e);
}
writeTypedField(b, indent, mt, name, f.id);
return;
}
b.append(indent).append(f.type).append(' ').append(name).append(" = ").append(f.id).append(";\n");
Expand All @@ -116,7 +132,11 @@ static void writeProtoGroup(StringBuilder b, XmlSchema.XmlGroup g,
String indent) {
String msgName = singularPascal(g.name);
b.append(indent).append("message ").append(msgName).append(" {\n");
for (XmlSchema.XmlField f : g.fields) writeProtoField(b, f, typeMap, compMap, enumMap, indent + " ");
try {
for (XmlSchema.XmlField f : g.fields) writeProtoField(b, f, typeMap, compMap, enumMap, indent + " ");
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("group \"" + g.name + "\": " + e.getMessage(), e);
}
b.append(indent).append("}\n");
b.append(indent).append("repeated ").append(msgName).append(' ').append(camelToSnake(g.name))
.append(" = ").append(g.id).append(";\n");
Expand All @@ -138,13 +158,32 @@ private static void writeTypedField(StringBuilder b, String indent, String[] mt,
* UTF-8, so a Latin-1 byte would fail the whole message (draft -01
* § Character Encoding, protowire#494, #173). As protowire-go's
* {@code xmltoproto.go} words it.
*
* <p>A {@code uint8} or {@code int8} array (length above 1) becomes
* {@code bytes} of the same {@code (sbe.length)}: the same N bytes in the
* block. An array of a wider primitive is refused, naming the type, since
* no proto field holds it and importing one element would misplace every
* later field in the block (draft -01 § Fixed-Length Arrays, #179).
*
* @throws IllegalArgumentException for an array of a primitive wider than a byte
*/
static String[] resolveTypeToProto(XmlSchema.XmlType t) {
String[] mt = resolveTypeToProto(t.primitiveType, t.length);
if ("char".equals(t.primitiveType) && !isUtf8Compatible(t.characterEncoding)) {
return new String[]{"bytes", mt[1], "SBE characterEncoding \"" + t.characterEncoding
+ "\" is not UTF-8 compatible, so this field is bytes."};
if ("char".equals(t.primitiveType)) {
String[] mt = resolveTypeToProto(t.primitiveType, t.length);
if (!isUtf8Compatible(t.characterEncoding)) {
return new String[]{"bytes", mt[1], "SBE characterEncoding \"" + t.characterEncoding
+ "\" is not UTF-8 compatible, so this field is bytes."};
}
return new String[]{mt[0], mt[1], ""};
}
if (t.length > 1) {
if ("uint8".equals(t.primitiveType) || "int8".equals(t.primitiveType)) {
return new String[]{"bytes", "(sbe.length) = " + t.length, ""};
}
throw new IllegalArgumentException("type \"" + t.name + "\" is an array of " + t.length + " "
+ t.primitiveType + ", which no proto field represents");
}
String[] mt = resolveTypeToProto(t.primitiveType, t.length);
return new String[]{mt[0], mt[1], ""};
}

Expand Down Expand Up @@ -196,6 +235,7 @@ public static String protoToXml(FileDescriptor... files) {
Integer version = SbeAnnotations.fileUint32(fd, SbeAnnotations.EXT_VERSION);

TreeSet<Integer> strLengths = new TreeSet<>();
TreeSet<Integer> bytesLengths = new TreeSet<>();
List<Descriptor> composites = new ArrayList<>();
Set<String> compSeen = new HashSet<>();
List<EnumDescriptor> enums = new ArrayList<>();
Expand All @@ -205,7 +245,7 @@ public static String protoToXml(FileDescriptor... files) {

for (Descriptor md : fd.getMessageTypes()) {
Integer tid = SbeAnnotations.messageUint32(md, SbeAnnotations.EXT_TEMPLATE_ID);
if (tid != null) collectXmlTypes(md, strLengths, composites, compSeen, enums, enumSeen);
if (tid != null) collectXmlTypes(md, strLengths, bytesLengths, composites, compSeen, enums, enumSeen);
else if (compSeen.add(md.getFullName())) composites.add(md);
}

Expand All @@ -227,7 +267,8 @@ public static String protoToXml(FileDescriptor... files) {
b.append(" <type name=\"blockLength\" primitiveType=\"uint16\"/>\n");
b.append(" <type name=\"numInGroup\" primitiveType=\"uint16\"/>\n");
b.append(" </composite>\n");
for (int l : strLengths) b.append(" <type name=\"str").append(l).append("\" primitiveType=\"char\" length=\"").append(l).append("\"/>\n");
for (int l : strLengths) b.append(" <type name=\"").append(strTypeName(l)).append("\" primitiveType=\"char\" length=\"").append(l).append("\"/>\n");
for (int l : bytesLengths) b.append(" <type name=\"").append(bytesTypeName(l)).append("\" primitiveType=\"uint8\" length=\"").append(l).append("\"/>\n");
for (EnumDescriptor ed : enums) writeXmlEnum(b, ed);
for (Descriptor md : composites) writeXmlComposite(b, md);
b.append(" </types>\n");
Expand All @@ -239,27 +280,32 @@ public static String protoToXml(FileDescriptor... files) {
return b.toString();
}

static void collectXmlTypes(Descriptor md, TreeSet<Integer> strLens, List<Descriptor> composites,
/** The {@code <types>} entry a string or bytes field of this fixed length refers to. */
static String strTypeName(int length) { return "str" + length; }
static String bytesTypeName(int length) { return "bytes" + length; }

static void collectXmlTypes(Descriptor md, TreeSet<Integer> strLens, TreeSet<Integer> bytesLens, List<Descriptor> composites,
Set<String> compSeen, List<EnumDescriptor> enums, Set<String> enumSeen) {
for (EnumDescriptor ed : md.getEnumTypes()) {
if (enumSeen.add(ed.getFullName())) enums.add(ed);
}
for (FieldDescriptor f : md.getFields()) {
if (f.getType() == FieldDescriptor.Type.STRING || f.getType() == FieldDescriptor.Type.BYTES) {
Integer len = SbeAnnotations.fieldUint32(f, SbeAnnotations.EXT_LENGTH);
if (len != null) strLens.add(len);
Integer len = SbeAnnotations.fieldUint32(f, SbeAnnotations.EXT_LENGTH);
if (len != null) {
if (f.getType() == FieldDescriptor.Type.STRING) strLens.add(len);
else if (f.getType() == FieldDescriptor.Type.BYTES) bytesLens.add(len);
}
if (f.getJavaType() == FieldDescriptor.JavaType.ENUM) {
EnumDescriptor ed = f.getEnumType();
if (enumSeen.add(ed.getFullName())) enums.add(ed);
}
if (f.getJavaType() == FieldDescriptor.JavaType.MESSAGE) {
if (f.isRepeated()) collectXmlTypes(f.getMessageType(), strLens, composites, compSeen, enums, enumSeen);
if (f.isRepeated()) collectXmlTypes(f.getMessageType(), strLens, bytesLens, composites, compSeen, enums, enumSeen);
else {
Descriptor sub = f.getMessageType();
if (compSeen.add(sub.getFullName())) {
composites.add(sub);
collectXmlTypes(sub, strLens, composites, compSeen, enums, enumSeen);
collectXmlTypes(sub, strLens, bytesLens, composites, compSeen, enums, enumSeen);
}
}
}
Expand Down Expand Up @@ -314,8 +360,7 @@ static void writeXmlField(StringBuilder b, FieldDescriptor fd, String indent) {
return;
}
SbeTypeInfo info = protoFieldToSbe(fd);
if (info.length > 0) b.append(indent).append("<field name=\"").append(fn).append("\" id=\"").append(fd.getNumber()).append("\" type=\"str").append(info.length).append("\"/>\n");
else b.append(indent).append("<field name=\"").append(fn).append("\" id=\"").append(fd.getNumber()).append("\" type=\"").append(info.xmlType).append("\"/>\n");
b.append(indent).append("<field name=\"").append(fn).append("\" id=\"").append(fd.getNumber()).append("\" type=\"").append(info.xmlType).append("\"/>\n");
}

static void writeXmlGroup(StringBuilder b, FieldDescriptor fd, String indent) {
Expand All @@ -327,14 +372,23 @@ static void writeXmlGroup(StringBuilder b, FieldDescriptor fd, String indent) {
b.append(indent).append("</group>\n");
}

/**
* An SBE type for a proto field. {@code xmlType} is what a {@code <field>}
* names: the primitive, or for a fixed-length array the {@code <types>}
* entry declared for its length.
*/
record SbeTypeInfo(String primitiveType, String xmlType, int length) {}

static SbeTypeInfo protoFieldToSbe(FieldDescriptor fd) {
String enc = SbeAnnotations.fieldString(fd, SbeAnnotations.EXT_ENCODING);
if (enc != null) return new SbeTypeInfo(enc, enc, 0);
// A bytes field is a uint8 array, not a char array: a char array
// imports back as a string (#179). Both are the same N bytes on the wire.
if (fd.getType() == FieldDescriptor.Type.STRING || fd.getType() == FieldDescriptor.Type.BYTES) {
Integer len = SbeAnnotations.fieldUint32(fd, SbeAnnotations.EXT_LENGTH);
return new SbeTypeInfo("char", "char", len == null ? 0 : len);
if (len == null || len == 0) return new SbeTypeInfo("char", "char", 0);
if (fd.getType() == FieldDescriptor.Type.BYTES) return new SbeTypeInfo("uint8", bytesTypeName(len), len);
return new SbeTypeInfo("char", strTypeName(len), len);
}
return switch (fd.getType()) {
case BOOL -> new SbeTypeInfo("uint8", "uint8", 0);
Expand Down
Loading
Loading