Repository navigation
pb: a PB codec for dataclasses, ported from protowire-go's encoding/pb - #53
Merged
Merged
Conversation
Fields are declared with Annotated[T, pb.Field(number, kind)]. marshal, unmarshal and the length-prefixed Encoder / Decoder write and read protowire-go v1.17.0's bytes, big numbers included, under Go's per-call limits. Go's encoding/pb tests are ported with their byte vectors; the conformance tests check the bytes against the protobuf runtime over protoc-compiled schemas, both ways.
As Go's harness decodes through its encoding/pb, into dataclasses that mirror adversarial.proto. All 29 pb corpus rows pass, with the same verdict as protowire-go's harness on each.
This was referenced Nov 8, 2026
| raise AssertionError(f"unexpected wire type {typ}") | ||
|
|
||
|
|
||
| def _submessage(b: bytes, num: int) -> bytes: |
…rough a local CodeQL flagged v != v (comparison of identical values) and raising an Optional attribute (illegal raise of None). Same behaviour.
Collaborator
Author
|
CI at |
trendvidia
pushed a commit
that referenced
this pull request
Nov 8, 2026
Four lines of the Unreleased section said what the entries after them undo: - check_decode --limit: "pb rows take none", but --format pb now takes every limit but MaxBytesLiteralLength, through protowire.pb (#53). - DatasetReader: rows parsed under the default max_bytes_literal_length "until protowire-cpp#160 is fixed". Fixed lists that fix. - the decoded message "stops at 100 levels" (#40), and a document with a literal at the depth limit "fails in upb" (#40). #51 fixed #40. - "the 16 that skip Python today": protowire#591 ran them. Measured for the new --limit wording: a limit pb has no setting for exits 2, and the same row under MaxMessageSize=1024 rejects with exit 1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python gets its own PB codec,
protowire.pb, as the owner decided on #36: a port of protowire-go'sencoding/pb(v1.17.0,ca79f5c). It writes and reads protowire-go's bytes, enforces Go's decode limits per call, and refuses to write a message nested pastmax_nesting_depth.check_decode --format pbnow goes through it, so Python can run the 16 corpus PB rows it skips today.Fixes #34. Part of #36: that issue closes when trendvidia/protowire#591 (Python off the PB rows'
skip) merges. The spec's gate checks out this repo'smain, so that PR has to wait for this one.The API (decided on #36: Annotated fields, a
pb.BigFloatclass, pure Python)intis int64 by default;pb.INT32/UINT32/UINT64/SINT32/SINT64/BIGINTname the others.floatis double by default, orpb.FLOAT.Optional[T]behaves as Go's pointer:Noneis absent.Optionalis always written, as a Go struct value is.decimal.Decimalispxf.Decimalandpb.BigFloatispxf.BigFloat.Two small additions the proposal didn't spell out:
unmarshaltakes a class as well as an instance; given an instance, it merges, as Go's does.Decoder.decodereturnsNoneat a clean end of stream. A truncated frame raisesEOFErrorand corrupt framing raisespb.FramingError; both leave the decoder failed, as Go'sErrUnexpectedEOF/ErrFramingCorruptdo.Evidence
test_pb*.py. Where Go's own test only round-trips a value, the Python test checks against the bytes Go'sMarshalwrites, recorded from a Go program.conf.Alland topxf/bignum.proto, as in Go'sconformance_test.goandbignum_conformance_test.go.453 passed in 1.62s(Python 3.15). The 253 pb tests alone give253 passedon Python 3.10.16, the oldest supported version. To check the assertions bite, I altered one expected vector and one expected message; the copy then failed 8 tests.pbrows of the spec'sMANIFEST.jsonl(spece6db25d) were run through this harness and through protowire-go'scheck_decode. 29 compared, 0 exit codes differ.cross_security_check.shwith the spec PR's manifest andREQUIRE_PORTS=go,python:Ran 152 (port, corpus) pairs across 2 ports, exit 0. On the spec'smainit runs 136 pairs. Python alone goes from 60 pairs to 76, the 16 rows.Differences from Go
None in the bytes or the verdicts measured above. Some edges can't carry over:
pb.INT32holding 2^31) is aValueErrorfrommarshal. Go's types cannot hold such a value.pb.BigFloathas no infinity, and Go'sMarshalrefuses an infinitebig.Floatanyway. A binary exponent above int32 is refused when apb.BigFloatis built; Go overflows to ±Inf there.ValueError. Go has no such ceiling, so raisingmax_nesting_depthpast a few hundred needssys.setrecursionlimitraised with it. The default of 100 is far inside the limit.UnmarshalOptions.Validator, since this package has no validation engine. Also not ported: Go-only test shapes such as a namedtype X bytefield, and the stream scratch-buffer reuse test. Each is noted where it would sit in the tests.Found on the way, filed: Go's harness mirror of
Treelackschildren, so a PB row nested through it would be skipped rather than depth-checked (trendvidia/protowire-go#285). No row does so today. This harness's mirror includes it.