Repository navigation
Description
pxf::DatasetReader (the streaming reader) differs from protowire-go's DatasetReader in four ways. The first one loses data silently. Measured on protowire-cpp v2.0.0 (cc25129) against protowire-go v1.16.1 (903b4a7), with the same inputs.
1. A cut-off or broken row ends the stream with no error
The spec's testdata/dataset/truncated-row-eof.pxf says such a row "MUST be rejected, by the streaming API as well as the materializing one: a row whose closing ")" never arrives is a decode error, not the end of the row sequence."
| Fixture (spec testdata/dataset, schema test.v1.AllTypes) | Go streaming Scan |
cpp streaming Scan (via protowire-python) |
cpp UnmarshalFull |
|---|---|---|---|
| truncated-row-eof.pxf | rejected after 1 row: 10:1: expected ',' or ')' in @dataset row, got EOF |
1 row, then a clean end | rejected, same message |
| truncated-row-then-rows.pxf | rejected after 1 row: 9:1: expected ',' or ')' in @dataset row, got ( |
1 row, then a clean end | rejected |
| junk-between-rows.pxf | rejected after 1 row: 8:1: @dataset directive cannot coexist with top-level field entries… |
1 row, then a clean end | rejected |
| rows-end-at-directive.pxf (control) | 2 rows | 2 rows | accepted |
Calling cpp's Scan directly gives the same: a row ("yy" with no ), or an unterminated string, returns OK with Done() true at every MaxMessageSize tried. Go returns 2:2: unterminated string or 3:1: expected ',' or ')' in @dataset row, got EOF.
So an interrupted append reads as a shorter, complete dataset.
2. No MaxMessageSize cap on bytes held while looking for a row's end
Go caps the bytes it holds while seeking a row boundary at MaxMessageSize (dataset input holds 57 bytes without a row boundary, MaxMessageSize=56). cpp's Next keeps pulling 4 KiB chunks into pending_ with no cap. A 10,000-byte row that never closes is buffered whole under max_message_size = 56, and then hits (1).
3. Scan binds a row under MaxMessageSize
Go's NewDatasetReader doc: "Scan binds a row under every limit but MaxMessageSize, as Unmarshal would bind the body the row stands for". The row's bytes were already capped as they were read, and the synthetic body is wider than the row, since it repeats the column names. cpp's Scan passes opts_ to BindCheckedRow unchanged. A one-row document with a 39-character column name and a 30-character string value:
| max_message_size | Go | cpp |
|---|---|---|
| 56 | accepted | input of 76 bytes exceeds MaxMessageSize=56 |
| 80 | accepted | accepted |
4. Rows are parsed without MaxBytesLiteralLength
Already filed as #160.
Done when
cpp's streaming reader gives Go's verdicts on these inputs, and the spec's truncated-row-* and junk-between-rows fixtures run through the streaming API in cpp's tests.
Found while checking protowire-python against protowire-go; tracked there by a linked issue.
Activity
bfrancojr commentedon Nov 7, 2026
Tracked in protowire-python by trendvidia/protowire-python#45, which is blocked on this.
pxf: DatasetReader ends a table only at the end of input or a directi…
pxf: DatasetReader ends a table only at the end of input or a directi…
release: cut v2.1.0
release: cut v2.1.0 (#175)
bfrancojr commentedon Nov 7, 2026
protowire-python now builds on v2.1.0 (trendvidia/protowire-python#46, merged as a1a1d70). Its
DatasetReaderrefuses the spec'struncated-row-eof,truncated-row-then-rowsandjunk-between-rowswith protowire-go v1.17.0's text and position, and still readsrows-end-at-directiveas two tables. Tests for all four are intests/test_pxf_dataset_reader.py. The same tests fail on a v2.0.0 build. Themax_message_sizecap shows up as Go'sdataset input holds N bytes without a row boundary. The Python-side half, wherescanbound rows undermax_message_size, was fixed in the same PR. That closes trendvidia/protowire-python#45.