Skip to content

pxf: DatasetReader positions count from the start of the stream (#161) - #165

Merged
trendvidia merged 1 commit into
mainfrom
dataset-row-position-161
Nov 7, 2026
Merged

trendvidia merged 1 commit into
mainfrom
dataset-row-position-161

Conversation

@bfrancojr

Copy link
Copy Markdown
Collaborator

Fixes #161

A refused dataset row now names its own line in the stream, as protowire-go does. Before, every row was parsed as if it started at 1:1, so in a long stream the error said 1:3 whatever row it came from.

The same cause reached further than the issue's error case. A row's pos and its cells' positions were also counted from the row, and so was the position of an error from Scan. All of them now match Go.

What changed

  • The reader keeps the stream position of the start of its buffer and moves it past every byte it consumes, as Go's base and posAt do.
  • Each row is parsed by a lexer that starts at the row's position in the stream, as Go's newParserAt does.
  • pxf::Lexer gains a constructor that takes that start position (line, column, offset). This adds to the public API. The existing constructor is unchanged.

Evidence

A Go probe run against protowire-go 903b4a7, with MaxBytesLiteralLength: 191, gives these lines. The new test DatasetReaderPositions.MatchProtowireGo expects them verbatim:

Case protowire-go output (= C++ now) C++ before
first row 2:3: bytes literal decodes to more than …=191 bytes 1:3: …
later row (after 2 good rows and a blank line, indented) rows pos=2:1@23, 3:1@29; error 5:5: … 1:1@0, 1:1@0; 1:3: …
rows on the header's line row pos=1:24@23, cell 1:26@25; error 1:34: … 1:1@0, 1:3@2; 1:3: …
a row's second line 2:5: … 2:5: … (matched already: that row starts on line 1)
arity 3:4: @dataset row has 1 cells, expected 2 (column count) 1:1: …
bad token 3:7: expected ',' or ')' in @dataset row, got integer 1:5: …
row 3001, about 18,000 bytes in (several 4 KiB reads) 3002:3: @dataset row has 2 cells, expected 1 (column count) not run before the fix
Scan, a column the type lacks 3:3: @dataset column "nope": google.protobuf.FieldDescriptorProto has no field "nope" not run before the fix
Scan, a cell of the wrong type 3:3: column number: expected integer for field "number" 1:1: …

With the fix stashed, 8 of the 9 cases fail. The exception is "a row's second line", for the reason in the table. With the fix, ctest passes 729/729 locally (macOS, Homebrew protobuf).

Classification

  • The change can be undone with a revert.
  • It adds one public constructor and changes no existing signature or type.
  • Callers see one difference: a row's positions are now counted from the start of the stream, not from the start of the row. That makes C++ match Go, which is the reference port.
  • It was measured against Go on the inputs above.

Caveat

This machine has clang-format 23, while the CI job runs 18. The new lines are clean under 23, and the CI clang-format job will check them under 18.

A row was parsed by a fresh lexer starting at 1:1, so a row's pos, its
cells' positions and every error from Next or Scan counted from the start
of that row. protowire-go starts the row's lexer at the row's place in
the stream (newParserAt with posAt(start)) and reports positions against
the whole stream.

The reader now keeps base_, the stream position of pending_[0], and moves
it past each byte it consumes. A row is parsed by a lexer that starts at
PosAt(start). pxf::Lexer gains a constructor taking that start: line and
column begin there and offsets add its offset. Input(), Mark and
RepositionTo still index the piece itself, and RepositionTo going back
returns to the start rather than to 1:1.

The test replays nine inputs and expects protowire-go 903b4a7's output
line for line: the first row, a later row, rows on the header's line, a
row's second line, an arity error, a bad token, a row 18000 bytes and
several reads in, and two Scan errors. Eight of the nine fail without the
fix; the row's second line matched already, since that row starts on
line 1.
@trendvidia
trendvidia merged commit 1673cd9 into main Nov 7, 2026
11 checks passed
@trendvidia
trendvidia deleted the dataset-row-position-161 branch November 7, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DatasetReader: a row's parse error gives a position within the row, not the stream as protowire-go does

2 participants