Skip to content

pb: a PB codec for dataclasses, ported from protowire-go's encoding/pb - #53

Merged
trendvidia merged 6 commits into
mainfrom
feat/36-pb-codec
Nov 8, 2026
Merged

trendvidia merged 6 commits into
mainfrom
feat/36-pb-codec

Conversation

@bfrancojr

@bfrancojr bfrancojr commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Python gets its own PB codec, protowire.pb, as the owner decided on #36: a port of protowire-go's encoding/pb (v1.17.0, ca79f5c). It writes and reads protowire-go's bytes, enforces Go's decode limits per call, and refuses to write a message nested past max_nesting_depth. check_decode --format pb now goes through it, so Python can run the 16 corpus PB rows it skips today.

Fixes #34. Part of #36: that issue closes when trendvidia/protowire#591 (Python off the PB rows' skip) merges. The spec's gate checks out this repo's main, so that PR has to wait for this one.

The API (decided on #36: Annotated fields, a pb.BigFloat class, pure Python)

@dataclass
class Order:
    id: Annotated[int, pb.Field(1, pb.UINT64)] = 0
    qty: Annotated[int, pb.Field(3, pb.SINT32)] = 0          # Go's `protowire:"3,zigzag"`
    price: Annotated[Optional[Decimal], pb.Field(4)] = None  # pxf.Decimal
    tags: Annotated[list[str], pb.Field(5)] = field(default_factory=list)

data = pb.marshal(order, max_nesting_depth=0)
order = pb.unmarshal(data, Order, max_message_size=0, max_nesting_depth=0,
                     max_numeric_literal_digits=0, max_repeated_count=0, max_big_float_exponent=0)
pb.Encoder(w).encode(order); pb.Decoder(r, max_frame_size=16 << 20).decode(Order)
  • int is int64 by default; pb.INT32/UINT32/UINT64/SINT32/SINT64/BIGINT name the others. float is double by default, or pb.FLOAT.
  • Optional[T] behaves as Go's pointer: None is absent.
  • A nested dataclass that is not Optional is always written, as a Go struct value is.
  • decimal.Decimal is pxf.Decimal and pb.BigFloat is pxf.BigFloat.

Two small additions the proposal didn't spell out:

  • unmarshal takes a class as well as an instance; given an instance, it merges, as Go's does.
  • Decoder.decode returns None at a clean end of stream. A truncated frame raises EOFError and corrupt framing raises pb.FramingError; both leave the decoder failed, as Go's ErrUnexpectedEOF / ErrFramingCorrupt do.

Evidence

  • Go's test files are ported with their byte vectors: 253 tests across test_pb*.py. Where Go's own test only round-trips a value, the Python test checks against the bytes Go's Marshal writes, recorded from a Go program.
  • Conformance: written bytes are compared, record by record, with what the protobuf runtime writes over the same schema compiled by protoc, and the runtime's bytes decode back to the same values. This applies to conf.All and to pxf/bignum.proto, as in Go's conformance_test.go and bignum_conformance_test.go.
  • Locally: the full suite gives 453 passed in 1.62s (Python 3.15). The 253 pb tests alone give 253 passed on Python 3.10.16, the oldest supported version. To check the assertions bite, I altered one expected vector and one expected message; the copy then failed 8 tests.
  • Corpus, row by row against Go: all 29 pb rows of the spec's MANIFEST.jsonl (spec e6db25d) were run through this harness and through protowire-go's check_decode. 29 compared, 0 exit codes differ.
  • Spec gate: cross_security_check.sh with the spec PR's manifest and REQUIRE_PORTS=go,python: Ran 152 (port, corpus) pairs across 2 ports, exit 0. On the spec's main it runs 136 pairs. Python alone goes from 60 pairs to 76, the 16 rows.

Differences from Go

None in the bytes or the verdicts measured above. Some edges can't carry over:

  • Python's ints have no width, so a value outside its declared type (pb.INT32 holding 2^31) is a ValueError from marshal. Go's types cannot hold such a value.
  • pb.BigFloat has no infinity, and Go's Marshal refuses an infinite big.Float anyway. A binary exponent above int32 is refused when a pb.BigFloat is built; Go overflows to ±Inf there.
  • A message nested past Python's recursion limit is a ValueError. Go has no such ceiling, so raising max_nesting_depth past a few hundred needs sys.setrecursionlimit raised with it. The default of 100 is far inside the limit.
  • Not ported: UnmarshalOptions.Validator, since this package has no validation engine. Also not ported: Go-only test shapes such as a named type X byte field, and the stream scratch-buffer reuse test. Each is noted where it would sit in the tests.

Found on the way, filed: Go's harness mirror of Tree lacks children, so a PB row nested through it would be skipped rather than depth-checked (trendvidia/protowire-go#285). No row does so today. This harness's mirror includes it.

Fields are declared with Annotated[T, pb.Field(number, kind)]. marshal,
unmarshal and the length-prefixed Encoder / Decoder write and read
protowire-go v1.17.0's bytes, big numbers included, under Go's per-call
limits. Go's encoding/pb tests are ported with their byte vectors; the
conformance tests check the bytes against the protobuf runtime over
protoc-compiled schemas, both ways.
As Go's harness decodes through its encoding/pb, into dataclasses that
mirror adversarial.proto. All 29 pb corpus rows pass, with the same
verdict as protowire-go's harness on each.
Comment thread src/protowire/pb.py Fixed
Comment thread src/protowire/pb.py Fixed
Comment thread src/protowire/pb.py Fixed
raise AssertionError(f"unexpected wire type {typ}")


def _submessage(b: bytes, num: int) -> bytes:
…rough a local

CodeQL flagged v != v (comparison of identical values) and raising an
Optional attribute (illegal raise of None). Same behaviour.
@bfrancojr

Copy link
Copy Markdown
Collaborator Author

CI at 9d523b1: 18/18 checks pass, CodeQL included. The first run's CodeQL failure was two findings in pb.py, v != v as a NaN test and raising an Optional attribute; 9d523b1 fixes both, with no behaviour change. In ubuntu-latest / py3.13 the new tests ran: 447 passed, 6 skipped, 453 in all, the same count as locally.

@trendvidia
trendvidia merged commit 4801f7e into main Nov 8, 2026
18 checks passed
@trendvidia
trendvidia deleted the feat/36-pb-codec branch November 8, 2026 09:11
trendvidia pushed a commit that referenced this pull request Nov 8, 2026
Four lines of the Unreleased section said what the entries after them
undo:
- check_decode --limit: "pb rows take none", but --format pb now takes
  every limit but MaxBytesLiteralLength, through protowire.pb (#53).
- DatasetReader: rows parsed under the default max_bytes_literal_length
  "until protowire-cpp#160 is fixed". Fixed lists that fix.
- the decoded message "stops at 100 levels" (#40), and a document with
  a literal at the depth limit "fails in upb" (#40). #51 fixed #40.
- "the 16 that skip Python today": protowire#591 ran them.

Measured for the new --limit wording: a limit pb has no setting for
exits 2, and the same row under MaxMessageSize=1024 rejects with exit 1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

producers: refuse a message nested past MaxNestingDepth (protowire#502)

3 participants