Per-call decoder limits: expose cpp's UnmarshalOptions limits and check_decode --limit

Problem

HARDENING § Mandatory limits says a caller can change each decoder limit per call. Python offers no way to do that. pxf.unmarshal, unmarshal_full, DatasetReader and sbe.Codec take no limit arguments, and scripts/check_decode.py says --limit is not implemented. So 12 rows of the adversarial corpus list python in skip, and Python is the only port skipping them: oversize-2kib (pxf, pb), giant-base64 (pxf), many-elements-16 (pxf, pb, sbe), each as a reject row and an accept row.

What exists upstream

protowire-cpp v2.0.0 has every knob these rows need. pxf::UnmarshalOptions has max_message_size, max_nesting_depth, max_bytes_literal_length, max_repeated_count and max_big_float_exponent. BindRow and DatasetReader::Create take options (cpp v2 changelog, Added). pxf::MarshalOptions::max_nesting_depth is #34's.

Not settled

The PB rows go through Python's stock google.protobuf decoder, not through protowire-cpp. Python cannot set MaxMessageSize or MaxRepeatedCount there. Measured on cpp cc25129: the store-only PB BigFloat rows (pb/bigfloat-*-exponent*, bigfloat-rounds-past-bound) are accepted where the corpus expects a reject. They list python in skip with no skip_reason. There are two ways to resolve this:

(a) Route Python's PB decode through cpp's pb::Unmarshal.
(b) Mark those rows with a skip_reason saying Python's PB path is the stock runtime. Other ports use such a reason for store-only decoders.

That choice belongs to the owner, and is not made here.

Done when

The PXF and SBE entry points take per-call limits that reach cpp's options, with a test per limit that the override both lowers and raises.
check_decode.py accepts --limit Name=N.
A protowire PR removes python from the skip of the PXF and SBE limit rows. It also removes python from pxf/bigfloat-huge-exponent.pxf and pxf/bigfloat-exponent-at-bound.pxf, which Python already passes on cpp v2 (measured).
The PB rows are handled the way the owner decides.

Related: #40. On the decode path the result reaches Python as PB bytes loaded by upb, which stops at 100 PB levels and cannot be raised. A per-call max_nesting_depth above 100 on unmarshal will hit that wall until #40 is settled.

Decided by the owner: the limits are keyword arguments named after cpp's fields (max_message_size, max_nesting_depth, max_numeric_literal_digits, max_bytes_literal_length, max_repeated_count, max_big_float_exponent), with 0 selecting the default. This matches pxf.marshal(max_nesting_depth=) from #41. sbe.Codec takes its two at construction, as cpp's CodecOptions does.

#42 implements the PXF/SBE limits and check_decode --limit. One limit is only partly honoured: trendvidia/protowire-cpp#160 (DatasetReader reads rows under the default max_bytes_literal_length). #42 pins that in an xfail test. Blocked by protowire-cpp#160 for that one edge; the rest doesn't depend on it.

The PXF and SBE half is merged in #42: per-call limits on pxf.unmarshal*, bind_row, DatasetReader.from_bytes, sbe.Codec, plus check_decode --limit. trendvidia/protowire#589 turns Python on for the eight pxf/sbe corpus rows that lower a limit. Locally the gate runs 136 pairs with that change and 128 without it, exit 0 both times.

This issue stays open on its PB question. The four pb rows with limits still skip Python, because stock google.protobuf has no limits to set, and check_decode exits 2 for them by design. You still need to decide: route PB decode through protowire-cpp, or mark these rows store-only with a skip_reason.

#34's PB-producer half and #36's PB rows are one question: should Python get its own PB codec? Checked against protowire-go (ca79f5c) and protowire-cpp (v2.1.0).

Go has two PB paths. One is the stock proto.Marshal / proto.Unmarshal, which has no protowire limits. The other is its own encoding/pb, a codec over Go structs with MarshalOptions.MaxNestingDepth and UnmarshalOptions{MaxMessageSize, MaxNestingDepth, MaxRepeatedCount, ...}. Go's check_decode --format pb runs the corpus PB rows through encoding/pb on hand-declared structs (scripts/check_decode/main.go:185).
cpp's pb.h is a port of that struct codec, as C++ templates over C++ structs. Python can't call it on a protobuf message. So "route Python's PB decode through cpp" (Per-call decoder limits: expose cpp's UnmarshalOptions limits and check_decode --limit #36's option a) isn't available. It would mean writing a new Python codec, over dataclasses or similar.
Python has only the stock path. It is the one port that skips pb/oversize-2kib and pb/many-elements-16: every other port has a PB decoder of its own.

The difference from Go is API surface, not behavior. Python's stock path behaves as Go's stock path does. Python lacks Go's second path.

Options

A. Port Go's encoding/pb to Python, over dataclasses, with its limits and its depth-refusing Marshal. A new public API. It closes producers: refuse a message nested past MaxNestingDepth (protowire#502) #34's PB half and Per-call decoder limits: expose cpp's UnmarshalOptions limits and check_decode --limit #36's PB rows by implementation.
B. Declare both not applicable: Python has no PB codec of its own. Close producers: refuse a message nested past MaxNestingDepth (protowire#502) #34's PB half. Give the PB rows a skip_reason ("no PB codec of its own: Python reads and writes PB through the stock google.protobuf runtime, which has no limits to set"), including the six pb/bigfloat-* rows that skip Python with no reason.

Recommendation: B. A Python user with protobuf messages uses google.protobuf anyway, and a struct codec in Python has no consumer asking for it.

Classification: spec metadata and docs, reversible. Python's lack of a codec is measured (it has no pb module). That nobody needs one is inferred.
What would flip it: a Python consumer decoding untrusted PB who needs MaxMessageSize or MaxRepeatedCount. That would be the case for A.

Decided by the owner (2026-11-07): port Go's encoding/pb to Python. Python gets its own PB codec, with Go's UnmarshalOptions limits and a Marshal that refuses past MaxNestingDepth. That closes this issue's PB half by implementation. An API proposal follows before the build.

API proposal: `protowire.pb`, a port of Go's `encoding/pb`

This is a new public API, so the shape is the owner's call before I build it. It follows Go's encoding/pb at v1.17.0 (ca79f5c): pb.go, options.go and stream.go, about 1,100 lines of non-test code.

What it is

A codec between protobuf binary and plain Python classes, with no .proto and no generated code. It's the counterpart of Go's struct-tag codec. As in Go:

marshal(obj, *, max_nesting_depth=0) -> bytes refuses a value nested past the limit (producers: refuse a message nested past MaxNestingDepth (protowire#502) #34's PB half).
unmarshal(data, obj, *, max_message_size=0, max_nesting_depth=0, max_numeric_literal_digits=0, max_repeated_count=0, max_big_float_exponent=0), with Go's UnmarshalOptions limits as keyword arguments, 0 for the default, as decided for Per-call decoder limits: expose cpp's UnmarshalOptions limits and check_decode --limit #36.
Encoder(file) / Decoder(file) for length-delimited streams, with Go's 16 MiB default frame cap.
Proto3 semantics: zero singular fields omitted, numeric repeated fields packed, both packed and unpacked accepted, strings checked for UTF-8.

How a class declares its fields (needs a decision)

Go reads the field number from a tag, and the wire type from the Go type (int32, uint64, float32, ...). A Python int or float doesn't say which wire type it is, so the declaration has to. Two shapes:

A. typing.Annotated on a dataclass (recommended)

from dataclasses import dataclass, field
from typing import Annotated
from protowire import pb

@dataclass
class Endpoint:
    path: Annotated[str, pb.Field(1)]
    port: Annotated[int, pb.Field(3, pb.INT32)]
    delta: Annotated[int, pb.Field(4, pb.SINT64)]       # Go's `protowire:"4,zigzag"`
    tags: Annotated[list[str], pb.Field(5)] = field(default_factory=list)
    inner: Annotated["Inner | None", pb.Field(6)] = None  # None = absent, like a nil pointer

B. dataclass field() metadata

    port: int = pb.field(3, pb.INT32, default=0)

A keeps the type and the wire type together and leaves defaults to plain dataclasses. B reads like the stdlib, but it hides the type in a call and must re-wrap dataclasses.field.

Defaults when nothing is given, mirroring Go's kinds: int → int64, float → double, bool, str, bytes, list[T] → repeated, dict[K, V] → map, a nested dataclass → submessage, T | None → presence. An int outside its declared range is a ValueError on marshal.

The big numbers (needs a decision)

Go maps big.Int → pxf.BigInt, big.Rat → pxf.Decimal, big.Float → pxf.BigFloat, with the same digit and exponent bounds.

BigInt: Annotated[int, pb.Field(n, pb.BIGINT)], since plain int means int64.
Decimal: decimal.Decimal, the obvious match.
BigFloat: Python has no arbitrary-precision binary float in the standard library. Options: (i) a small pb.BigFloat(mantissa, exponent, prec, negative) value class, holding exactly what the wire holds, with no arithmetic; (ii) depend on gmpy2 or mpmath. Recommendation: (i), so the package gains no dependency. Arithmetic can be added later, or a conversion to mpmath offered.

Implementation

Pure Python. cpp's port (pb.h) is templates over C++ structs, so it can't be called on a Python class (see above). Speed won't match the native PXF path. The limits and the wire format are what the corpus checks. check_decode --format pb would decode the corpus rows into hand-declared classes, as Go's does (scripts/check_decode/main.go:185). A spec PR then removes python from the PB rows' skip.

Tests

Go's pb tests carry their expected bytes. I'd port the conformance, limits, depth, UTF-8, bignum and stream tests with Go's vectors, so a byte-for-byte difference from Go fails a test.

Classification

New public API: foundational, and reversible only by a deprecation once released. The behaviour is Go's, measured against Go's test vectors as they're ported. The choices above (declaration shape, BigFloat type, pure Python) are design, not measurement.
What would flip the recommendations: for A vs B, a type-checker or IDE that handles one shape badly, which I'd check on a prototype. For BigFloat (i), a consumer who needs to compute with the values.

Decided by the owner (2026-11-07): the recommendations in the API proposal. Fields are declared with typing.Annotated[T, pb.Field(n, ...)] on dataclasses (shape A). A BigFloat is a pb.BigFloat value class holding what the wire holds, with no new dependency (option i). The codec is pure Python. Building it now.

Implemented in #53 (protowire.pb, with Go's limits and check_decode --format pb through it). The remaining piece is trendvidia/protowire#591, which drops python from the 16 PB rows' skip and has to merge after #53. This issue closes when #591 merges.

corpus: run python on the pb rows (#591)

Done. Each done-when item, checked against what merged:

PXF and SBE take per-call limits, with a lower-and-raise test per limit: pxf, sbe: take the decode limits per call; check_decode --limit #42.
check_decode.py --limit Name=N: pxf, sbe: take the decode limits per call; check_decode --limit #42.
The protowire PR dropping python from the PXF/SBE limit rows and the two pxf/bigfloat-* rows: corpus: run python on the eight pxf/sbe rows that lower a limit protowire#589. On protowire's main manifest, no row lists python in skip, all 12 limit rows included.
The PB rows, handled as the owner decided (Python gets its own PB codec): protowire.pb in pb: a PB codec for dataclasses, ported from protowire-go's encoding/pb #53, then corpus: run python on the pb rows protowire#591. Its hardening-corpus check ran 520 pairs across 7 ports with no failures at head ad94a65 with protowire-python 4801f7e.

The DatasetReader edge pinned in #42 is fixed too (trendvidia/protowire-cpp#160 is closed). #54 removes the changelog caveats this work left stale.

Navigation Menu

Repository navigation

Per-call decoder limits: expose cpp's UnmarshalOptions limits and check_decode --limit #36

Problem

What exists upstream

Not settled

Done when

API proposal: `protowire.pb`, a port of Go's `encoding/pb`

What it is

How a class declares its fields (needs a decision)

The big numbers (needs a decision)

Implementation

Tests

Classification

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Per-call decoder limits: expose cpp's UnmarshalOptions limits and check_decode --limit #36

Description

Problem

What exists upstream

Not settled

Done when

Activity

bfrancojr commented on Nov 6, 2026

bfrancojr commented on Nov 6, 2026

bfrancojr commented on Nov 6, 2026

bfrancojr commented on Nov 6, 2026

bfrancojr commented on Nov 8, 2026

bfrancojr commented on Nov 8, 2026

bfrancojr commented on Nov 8, 2026

API proposal: protowire.pb, a port of Go's encoding/pb

What it is

How a class declares its fields (needs a decision)

The big numbers (needs a decision)

Implementation

Tests

Classification

bfrancojr commented on Nov 8, 2026

bfrancojr commented on Nov 8, 2026

bfrancojr commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

API proposal: `protowire.pb`, a port of Go's `encoding/pb`