Skip to content

DatasetReader: a cut-off row ends the stream silently, and its limits differ from protowire-go's #164

Closed

Description

@bfrancojr

pxf::DatasetReader (the streaming reader) differs from protowire-go's DatasetReader in four ways. The first one loses data silently. Measured on protowire-cpp v2.0.0 (cc25129) against protowire-go v1.16.1 (903b4a7), with the same inputs.

1. A cut-off or broken row ends the stream with no error

The spec's testdata/dataset/truncated-row-eof.pxf says such a row "MUST be rejected, by the streaming API as well as the materializing one: a row whose closing ")" never arrives is a decode error, not the end of the row sequence."

Fixture (spec testdata/dataset, schema test.v1.AllTypes) Go streaming Scan cpp streaming Scan (via protowire-python) cpp UnmarshalFull
truncated-row-eof.pxf rejected after 1 row: 10:1: expected ',' or ')' in @dataset row, got EOF 1 row, then a clean end rejected, same message
truncated-row-then-rows.pxf rejected after 1 row: 9:1: expected ',' or ')' in @dataset row, got ( 1 row, then a clean end rejected
junk-between-rows.pxf rejected after 1 row: 8:1: @dataset directive cannot coexist with top-level field entries… 1 row, then a clean end rejected
rows-end-at-directive.pxf (control) 2 rows 2 rows accepted

Calling cpp's Scan directly gives the same: a row ("yy" with no ), or an unterminated string, returns OK with Done() true at every MaxMessageSize tried. Go returns 2:2: unterminated string or 3:1: expected ',' or ')' in @dataset row, got EOF.

So an interrupted append reads as a shorter, complete dataset.

2. No MaxMessageSize cap on bytes held while looking for a row's end

Go caps the bytes it holds while seeking a row boundary at MaxMessageSize (dataset input holds 57 bytes without a row boundary, MaxMessageSize=56). cpp's Next keeps pulling 4 KiB chunks into pending_ with no cap. A 10,000-byte row that never closes is buffered whole under max_message_size = 56, and then hits (1).

3. Scan binds a row under MaxMessageSize

Go's NewDatasetReader doc: "Scan binds a row under every limit but MaxMessageSize, as Unmarshal would bind the body the row stands for". The row's bytes were already capped as they were read, and the synthetic body is wider than the row, since it repeats the column names. cpp's Scan passes opts_ to BindCheckedRow unchanged. A one-row document with a 39-character column name and a 30-character string value:

max_message_size Go cpp
56 accepted input of 76 bytes exceeds MaxMessageSize=56
80 accepted accepted

4. Rows are parsed without MaxBytesLiteralLength

Already filed as #160.

Done when

cpp's streaming reader gives Go's verdicts on these inputs, and the spec's truncated-row-* and junk-between-rows fixtures run through the streaming API in cpp's tests.

Found while checking protowire-python against protowire-go; tracked there by a linked issue.

Activity

  1. bfrancojr commented on Nov 7, 2026

    @bfrancojr
    CollaboratorAuthor

    Tracked in protowire-python by trendvidia/protowire-python#45, which is blocked on this.

  2. bfrancojr commented on Nov 7, 2026

    @bfrancojr
    CollaboratorAuthor

    protowire-python now builds on v2.1.0 (trendvidia/protowire-python#46, merged as a1a1d70). Its DatasetReader refuses the spec's truncated-row-eof, truncated-row-then-rows and junk-between-rows with protowire-go v1.17.0's text and position, and still reads rows-end-at-directive as two tables. Tests for all four are in tests/test_pxf_dataset_reader.py. The same tests fail on a v2.0.0 build. The max_message_size cap shows up as Go's dataset input holds N bytes without a row boundary. The Python-side half, where scan bound rows under max_message_size, was fixed in the same PR. That closes trendvidia/protowire-python#45.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions