Repository navigation
DatasetReader: a cut-off last row reads as the end of the dataset; limits differ from protowire-go #45
Description
pxf.DatasetReader doesn't match protowire-go's streaming reader. The worst case: a dataset whose last row was cut off reads as a shorter, complete dataset, with no error.
Measured (main df004ec, protowire-cpp v2.0.0)
The spec's testdata/dataset fixtures through DatasetReader.from_bytes(...).scan(...), schema test.v1.AllTypes:
| Fixture | Spec / Go streaming | Python streaming | Python unmarshal_full |
|---|---|---|---|
| truncated-row-eof.pxf | reject | 1 row, then scan returns False |
reject |
| truncated-row-then-rows.pxf | reject | 1 row, then False | reject |
| junk-between-rows.pxf | reject | 1 row, then False | reject |
| rows-end-at-directive.pxf | accept, 2 rows | 2 rows | accept |
Causes
- protowire-cpp's reader (DatasetReader: a cut-off row ends the stream silently, and its limits differ from protowire-go's protowire-cpp#164): it ends the stream at a broken row instead of failing, holds bytes while seeking a row's end with no MaxMessageSize cap, and parses rows without MaxBytesLiteralLength (DatasetReader ignores UnmarshalOptions::max_bytes_literal_length when it parses a row protowire-cpp#160, pinned here as an xfail).
- This port's
scanpassesmax_message_sizetobind_rowfor every row. Go binds a row "under every limit but MaxMessageSize", because the synthetic body repeats the column names and is wider than the row. This half can be fixed here. It belongs with pxf.bind_row promises (pxf.default) and (pxf.required) but applies neither #24, which decides howbind_rowbinds (its own code or cpp'sBindRow).
Done when
- The four fixtures above give Go's verdicts through
DatasetReader, with a test that runs them (after protowire-cpp#164 is fixed and the pin moves). scandoesn't applymax_message_sizeto the bound row, with a test.
Blocked by trendvidia/protowire-cpp#164 for the first item.
Activity
bfrancojr commentedon Nov 7, 2026
The protowire-cpp half of this is fixed on cpp's main. It is not in a cpp release yet; v2.0.0 predates it.
Parsegives. The bytes held while looking for a row's end are capped atmax_message_size.Scanbinds a row under the defaultmax_message_size, as Go does.MaxBytesLiteralLengthincluded.Evidence on cpp:
test/pxf_corpus_test.ccstreams every spec dataset fixture throughDatasetReaderand requires the same verdict and error asParse.truncated-row-eof,truncated-row-then-rowsandjunk-between-rowsare now refused there.DatasetReaderEnds.MatchProtowireGochecks 590 cut points against a table generated from protowire-go v1.16.1.What is left here:
DatasetReader.from_bytes(...).scan(...)and drop the #160 xfail.scanhalf (not applyingmax_message_sizeto the bound row) is still this port's to fix. That is unchanged.bfrancojr commentedon Nov 7, 2026
protowire-cpp v2.1.0 is released (tag v2.1.0 → a4977b2) and carries trendvidia/protowire-cpp#167 (#164), #162 (#160) and the rest of the DatasetReader fixes. The first item here can go ahead now: move the cpp pin to v2.1.0, run the four fixtures through
DatasetReader.from_bytes(...).scan(...), and drop the #160 xfail. Thescanhalf is unchanged and still this port's to fix.