Skip to content

Build on protowire-cpp v2.1.0; DatasetReader refuses a broken row - #46

Merged
trendvidia merged 2 commits into
mainfrom
cpp-v2.1.0
Nov 7, 2026
Merged

trendvidia merged 2 commits into
mainfrom
cpp-v2.1.0

Conversation

@bfrancojr

Copy link
Copy Markdown
Collaborator

Moves the protowire-cpp pin to v2.1.0 so pxf.DatasetReader reads a @dataset stream as protowire-go does. Also stops scan applying max_message_size to the row it binds. Fixes #45.

What changes for callers

  • A row cut short is an error, not the end of the dataset. On v2.0.0 a last row whose ) never came, a broken row followed by more rows, or junk between rows read as one clean row and then EOF. An interrupted append looked like a shorter, complete dataset. Now each gives the error unmarshal_full gives, at the same position.
  • Rows are read under the reader's max_bytes_literal_length (protowire-cpp#160). The strict xfail pinned for it is removed.
  • The reader caps the bytes it holds at max_message_size. Before, a row that never closed was buffered whole.
  • scan binds each row under every limit except max_message_size, as Go does. The bound body repeats the column names, so it is wider than the row, and a row the reader accepted could still be refused when bound.

Evidence

The four spec fixtures (testdata/dataset/ at protowire dd8e221), streamed through protowire-go v1.17.0's DatasetReader and through this branch:

Fixture Go v1.17.0 This branch cpp v2.0.0
truncated-row-eof 1 row, then 10:1: expected ',' or ')' in @dataset row, got EOF same text and position 1 row, then EOF
truncated-row-then-rows 1 row, then 9:1: expected ',' or ')' in @dataset row, got ( same 1 row, then EOF
junk-between-rows 1 row, then 8:1: @dataset directive cannot coexist with top-level field entries; … same 1 row, then EOF
rows-end-at-directive 2 tables, 2 rows + 1 row same same

They are copied into tests/test_pxf_dataset_reader.py verbatim (checked byte-equal to the spec files), because CI has no spec checkout.

The scan limit. Table: 40 int32 columns with 200-character names, 3 rows. Input 8457 bytes, row 121 bytes, bound body 8200 bytes. At max_message_size=8192:

  • Go: reads all 3 rows.
  • This branch: reads all 3 rows.
  • main's scan on cpp v2.1.0: pxf.unmarshal: 1:1: input of 8200 bytes exceeds MaxMessageSize=8192.

A small table can't show this. The reader's cap counts the whole held buffer, header included, and it reads 4 KiB at a time. So under one read, header plus row is always wider than the bound body. That is why the test table is so wide.

Tests, macOS, Python 3.15, local build against cpp v2.1.0 (a4977b2):

  • Branch: 186 passed.
  • First commit alone (main's pxf.py): 172 passed, 13 skipped. The 13 skips are tests that need the spec checkout next to the repo.
  • New tests against a build of cpp v2.0.0 (tree identical to the tag): 5 fail, as they should. These are the 3 broken-row fixtures, the #160 test and the scan test.
  • New scan test against main's scan on v2.1.0: fails, as it should.

HARDENING corpus (cross_security_check.sh at protowire dd8e221, REQUIRE_PORTS=python,cpp, other ports skipped): Ran 136 (port, corpus) pairs across 2 ports. ✓ All ports conform. Python alone: Ran 60 … ✓. That is the same 136 as the last run on v2.0.0. An all-ports run also fails 3 Rust rows, already filed as protowire-rust#86 and #87.

Not covered here

  • CI covers Linux, Windows and Python 3.10–3.13. Locally I ran only macOS on 3.15.
  • v2.1.0's other changes (comments kept in the AST, Scan/BindRow error positions in cpp) don't reach Python. This port exposes no AST or formatter, and bind_row is its own code.
  • Whether bind_row should route through cpp's BindRow (pxf.bind_row promises (pxf.default) and (pxf.required) but applies neither #24) is still open. This PR only changes which limits scan passes to it.

v2.1.0's streaming reader ends a @dataset table only at the end of the
input or at the next directive (protowire-cpp#164). On v2.0.0 a row cut
short, or junk between rows, read as a shorter, complete dataset. The
spec's truncated-row-eof, truncated-row-then-rows and junk-between-rows
fixtures now give protowire-go v1.17.0's error, text and position, and
rows-end-at-directive still reads as two tables. They are copied in
verbatim; CI has no spec checkout.

Rows are read under the caller's max_bytes_literal_length
(protowire-cpp#160), so the strict xfail pinned for it is removed.
…size

protowire-go binds a row under every limit but MaxMessageSize: the body
a row is bound as repeats the column names, so it is wider than the row,
whose bytes the reader has already capped. scan passed the reader's
max_message_size on to bind_row, so a row the reader accepted could be
refused when bound. Measured on a 40-column table with 200-character
names: at max_message_size=8192 Go reads all three rows, and this port
failed the first with 'input of 8200 bytes exceeds MaxMessageSize=8192'.

Fixes #45
@bfrancojr

Copy link
Copy Markdown
Collaborator Author

CI on deaa6ec: all 12 test legs (Linux, macOS, Windows × py3.10–3.13), the 3 cibuildwheel smoke builds and CodeQL (cpp, python) pass. Test count read from the ubuntu py3.10 and windows py3.13 logs: 181 passed, 5 skipped, which is 186, the local count. The 5 skips are the SBE view tests that need a spec checkout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DatasetReader: a cut-off last row reads as the end of the dataset; limits differ from protowire-go

2 participants