Skip to content

pxf, sbe: take the decode limits per call; check_decode --limit - #42

Merged
trendvidia merged 2 commits into
mainfrom
decode-limits
Nov 6, 2026
Merged

trendvidia merged 2 commits into
mainfrom
decode-limits

Conversation

@bfrancojr

Copy link
Copy Markdown
Collaborator

Part of #36. It covers the PXF and SBE limits and the harness flag. #36 stays open for the spec-repo PR that unskips the 8 PXF/SBE limit rows (I'll open it once this merges) and for your PB-rows decision.

What changes for callers

  • pxf.unmarshal, unmarshal_full, unmarshal_bytes, unmarshal_full_bytes, bind_row and DatasetReader.from_bytes take max_message_size, max_nesting_depth, max_numeric_literal_digits, max_bytes_literal_length, max_repeated_count and max_big_float_exponent. They are keyword arguments, as you decided on Per-call decoder limits: expose cpp's UnmarshalOptions limits and check_decode --limit #36, passed to protowire-cpp's pxf::UnmarshalOptions. 0 keeps the default; negative is a ValueError.
  • sbe.Codec(...) and Codec.from_message(...) take max_message_size and max_repeated_count once, for every call, as cpp's sbe::CodecOptions does.
  • DatasetReader.scan binds each row under the reader's limits, so a cell the reader accepted under a raised limit isn't refused at bind time.
  • scripts/check_decode.py --limit MaxMessageSize=1024 (Go names, as the manifest spells them). pxf takes all six, sbe two, pb-render MaxNestingDepth. A limit the format can't apply exits 2 rather than running the row without it. pb takes none: google.protobuf has no such settings.

Evidence

Built against protowire-cpp v2.0.0 (tree of cc25129 = 4e96aba):

  • pytest: 180 passed, 1 xfailed. That is main's 152, plus 24 in test_pxf_decode_limits.py and 4 in test_sbe_limits.py.
    • Each PXF limit is shown lowered: refused just under the input, accepted at it.
    • Five are also shown raised: the default refuses the input, the raised limit accepts it. The two 64 MiB cases (message size, bytes literal) run in under a second, peaking near 570 MB.
    • max_nesting_depth is raised through lists (depth 120, 60 protobuf levels), because a chain of submessages past 100 hits the protobuf runtime's own limit (pxf.unmarshal: a document at the depth limit fails in upb when a literal sits at the bottom #40).
  • check_decode on every manifest row with limits: all 8 pxf/sbe rows match their expect. The 4 pb rows exit 2, --limit ... is not supported for pb.
  • cross_security_check.sh, REQUIRE_PORTS=python, spec at corpus: run python on four rows it passes on protowire-cpp v2.0.0 protowire#587 with python removed locally from those 8 rows' skip: exit 0, Ran 136 (port, corpus) pairs. Without the local edit it's 128.

Not covered, and why

  • max_repeated_count is not tested raised: its default is 64 Mi elements.
  • SBE limits are tested lowered only. A group count is a uint16 on the wire, far below the 64 Mi default, and an Order message is a few dozen bytes.

Found on the way, filed

Still open (owner)

HARDENING.md § Mandatory limits says a caller can change each decoder limit
per call. The PXF decoders, bind_row and DatasetReader.from_bytes now take
max_message_size, max_nesting_depth, max_numeric_literal_digits,
max_bytes_literal_length, max_repeated_count and max_big_float_exponent as
keyword arguments, passed to pxf::UnmarshalOptions; 0 keeps the default.
sbe.Codec takes max_message_size and max_repeated_count at construction,
as sbe::CodecOptions does. DatasetReader.scan binds under the reader's
limits.

The reader parses rows under the default bytes-literal limit
(protowire-cpp#160), pinned in an xfail test.
The corpus rows that lower a limit pass it as --limit with the Go name.
pxf rows take all six, sbe rows MaxMessageSize and MaxRepeatedCount, and
pb-render MaxNestingDepth. A limit the format cannot apply exits 2, since
running the row without it would not test the limit. pb rows take none:
the protobuf runtime has no such settings.
@bfrancojr

Copy link
Copy Markdown
Collaborator Author

CI 18/18 pass on 9df29f7 (run 42276895455). protowire-cpp at 4e96aba release: cut v2.0.0 (#155).

@trendvidia
trendvidia merged commit d20bb77 into main Nov 6, 2026
18 checks passed
@trendvidia
trendvidia deleted the decode-limits branch November 6, 2026 20:03
trendvidia pushed a commit to trendvidia/protowire that referenced this pull request Nov 7, 2026
protowire-python's check_decode takes --limit since trendvidia/protowire-python#42,
and passes all eight. The four pb rows with limits keep python in skip:
the stock protobuf runtime has no such settings, so its harness exits 2.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants