Repository navigation
pxf, sbe: take the decode limits per call; check_decode --limit - #42
Merged
Merged
Conversation
HARDENING.md § Mandatory limits says a caller can change each decoder limit per call. The PXF decoders, bind_row and DatasetReader.from_bytes now take max_message_size, max_nesting_depth, max_numeric_literal_digits, max_bytes_literal_length, max_repeated_count and max_big_float_exponent as keyword arguments, passed to pxf::UnmarshalOptions; 0 keeps the default. sbe.Codec takes max_message_size and max_repeated_count at construction, as sbe::CodecOptions does. DatasetReader.scan binds under the reader's limits. The reader parses rows under the default bytes-literal limit (protowire-cpp#160), pinned in an xfail test.
The corpus rows that lower a limit pass it as --limit with the Go name. pxf rows take all six, sbe rows MaxMessageSize and MaxRepeatedCount, and pb-render MaxNestingDepth. A limit the format cannot apply exits 2, since running the row without it would not test the limit. pb rows take none: the protobuf runtime has no such settings.
Collaborator
Author
|
CI 18/18 pass on 9df29f7 (run 42276895455). protowire-cpp at
|
This was referenced Nov 6, 2026
trendvidia
pushed a commit
to trendvidia/protowire
that referenced
this pull request
Nov 7, 2026
protowire-python's check_decode takes --limit since trendvidia/protowire-python#42, and passes all eight. The four pb rows with limits keep python in skip: the stock protobuf runtime has no such settings, so its harness exits 2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #36. It covers the PXF and SBE limits and the harness flag. #36 stays open for the spec-repo PR that unskips the 8 PXF/SBE limit rows (I'll open it once this merges) and for your PB-rows decision.
What changes for callers
pxf.unmarshal,unmarshal_full,unmarshal_bytes,unmarshal_full_bytes,bind_rowandDatasetReader.from_bytestakemax_message_size,max_nesting_depth,max_numeric_literal_digits,max_bytes_literal_length,max_repeated_countandmax_big_float_exponent. They are keyword arguments, as you decided on Per-call decoder limits: expose cpp's UnmarshalOptions limits and check_decode --limit #36, passed to protowire-cpp'spxf::UnmarshalOptions. 0 keeps the default; negative is aValueError.sbe.Codec(...)andCodec.from_message(...)takemax_message_sizeandmax_repeated_countonce, for every call, as cpp'ssbe::CodecOptionsdoes.DatasetReader.scanbinds each row under the reader's limits, so a cell the reader accepted under a raised limit isn't refused at bind time.scripts/check_decode.py --limit MaxMessageSize=1024(Go names, as the manifest spells them). pxf takes all six, sbe two, pb-renderMaxNestingDepth. A limit the format can't apply exits 2 rather than running the row without it. pb takes none:google.protobufhas no such settings.Evidence
Built against protowire-cpp v2.0.0 (tree of cc25129 = 4e96aba):
180 passed, 1 xfailed. That is main's 152, plus 24 intest_pxf_decode_limits.pyand 4 intest_sbe_limits.py.max_nesting_depthis raised through lists (depth 120, 60 protobuf levels), because a chain of submessages past 100 hits the protobuf runtime's own limit (pxf.unmarshal: a document at the depth limit fails in upb when a literal sits at the bottom #40).check_decodeon every manifest row withlimits: all 8 pxf/sbe rows match theirexpect. The 4 pb rows exit 2,--limit ... is not supported for pb.cross_security_check.sh,REQUIRE_PORTS=python, spec at corpus: run python on four rows it passes on protowire-cpp v2.0.0 protowire#587 withpythonremoved locally from those 8 rows'skip: exit 0,Ran 136 (port, corpus) pairs. Without the local edit it's 128.Not covered, and why
max_repeated_countis not tested raised: its default is 64 Mi elements.Found on the way, filed
DatasetReaderparses rows under the defaultmax_bytes_literal_length, not the caller's; protowire-go applies the caller's. That's pinned here as a strictxfail, so the test flips when cpp is fixed.scanstill applies the limit when it binds.b"…"literal's decoded length aschars / 4 * 3, so a padded literal decoding to exactly the limit is refused. It's cross-port and needs a spec decision, so it isn't fixed here. The tests here use unpadded sizes.Still open (owner)
skip_reason.