Repository navigation
Build on protowire-cpp v2.1.0; DatasetReader refuses a broken row - #46
Merged
Merged
Conversation
v2.1.0's streaming reader ends a @dataset table only at the end of the input or at the next directive (protowire-cpp#164). On v2.0.0 a row cut short, or junk between rows, read as a shorter, complete dataset. The spec's truncated-row-eof, truncated-row-then-rows and junk-between-rows fixtures now give protowire-go v1.17.0's error, text and position, and rows-end-at-directive still reads as two tables. They are copied in verbatim; CI has no spec checkout. Rows are read under the caller's max_bytes_literal_length (protowire-cpp#160), so the strict xfail pinned for it is removed.
…size protowire-go binds a row under every limit but MaxMessageSize: the body a row is bound as repeats the column names, so it is wider than the row, whose bytes the reader has already capped. scan passed the reader's max_message_size on to bind_row, so a row the reader accepted could be refused when bound. Measured on a 40-column table with 200-character names: at max_message_size=8192 Go reads all three rows, and this port failed the first with 'input of 8200 bytes exceeds MaxMessageSize=8192'. Fixes #45
Collaborator
Author
|
CI on deaa6ec: all 12 test legs (Linux, macOS, Windows × py3.10–3.13), the 3 cibuildwheel smoke builds and CodeQL (cpp, python) pass. Test count read from the ubuntu py3.10 and windows py3.13 logs: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the protowire-cpp pin to v2.1.0 so
pxf.DatasetReaderreads a@datasetstream as protowire-go does. Also stopsscanapplyingmax_message_sizeto the row it binds. Fixes #45.What changes for callers
)never came, a broken row followed by more rows, or junk between rows read as one clean row and then EOF. An interrupted append looked like a shorter, complete dataset. Now each gives the errorunmarshal_fullgives, at the same position.max_bytes_literal_length(protowire-cpp#160). The strict xfail pinned for it is removed.max_message_size. Before, a row that never closed was buffered whole.scanbinds each row under every limit exceptmax_message_size, as Go does. The bound body repeats the column names, so it is wider than the row, and a row the reader accepted could still be refused when bound.Evidence
The four spec fixtures (
testdata/dataset/at protowire dd8e221), streamed through protowire-go v1.17.0'sDatasetReaderand through this branch:10:1: expected ',' or ')' in @dataset row, got EOF9:1: expected ',' or ')' in @dataset row, got (8:1: @dataset directive cannot coexist with top-level field entries; …They are copied into
tests/test_pxf_dataset_reader.pyverbatim (checked byte-equal to the spec files), because CI has no spec checkout.The
scanlimit. Table: 40 int32 columns with 200-character names, 3 rows. Input 8457 bytes, row 121 bytes, bound body 8200 bytes. Atmax_message_size=8192:scanon cpp v2.1.0:pxf.unmarshal: 1:1: input of 8200 bytes exceeds MaxMessageSize=8192.A small table can't show this. The reader's cap counts the whole held buffer, header included, and it reads 4 KiB at a time. So under one read, header plus row is always wider than the bound body. That is why the test table is so wide.
Tests, macOS, Python 3.15, local build against cpp v2.1.0 (a4977b2):
186 passed.pxf.py):172 passed, 13 skipped. The 13 skips are tests that need the spec checkout next to the repo.scantest.scantest against main'sscanon v2.1.0: fails, as it should.HARDENING corpus (
cross_security_check.shat protowire dd8e221,REQUIRE_PORTS=python,cpp, other ports skipped):Ran 136 (port, corpus) pairs across 2 ports. ✓ All ports conform. Python alone:Ran 60 … ✓. That is the same 136 as the last run on v2.0.0. An all-ports run also fails 3 Rust rows, already filed as protowire-rust#86 and #87.Not covered here
Scan/BindRowerror positions in cpp) don't reach Python. This port exposes no AST or formatter, andbind_rowis its own code.bind_rowshould route through cpp'sBindRow(pxf.bind_row promises (pxf.default) and (pxf.required) but applies neither #24) is still open. This PR only changes which limitsscanpasses to it.