Repository navigation
pxf: DatasetReader positions count from the start of the stream (#161) - #165
Merged
Merged
Conversation
A row was parsed by a fresh lexer starting at 1:1, so a row's pos, its cells' positions and every error from Next or Scan counted from the start of that row. protowire-go starts the row's lexer at the row's place in the stream (newParserAt with posAt(start)) and reports positions against the whole stream. The reader now keeps base_, the stream position of pending_[0], and moves it past each byte it consumes. A row is parsed by a lexer that starts at PosAt(start). pxf::Lexer gains a constructor taking that start: line and column begin there and offsets add its offset. Input(), Mark and RepositionTo still index the piece itself, and RepositionTo going back returns to the start rather than to 1:1. The test replays nine inputs and expects protowire-go 903b4a7's output line for line: the first row, a later row, rows on the header's line, a row's second line, an arity error, a bad token, a row 18000 bytes and several reads in, and two Scan errors. Eight of the nine fail without the fix; the row's second line matched already, since that row starts on line 1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #161
A refused dataset row now names its own line in the stream, as protowire-go does. Before, every row was parsed as if it started at
1:1, so in a long stream the error said1:3whatever row it came from.The same cause reached further than the issue's error case. A row's
posand its cells' positions were also counted from the row, and so was the position of an error fromScan. All of them now match Go.What changed
baseandposAtdo.newParserAtdoes.pxf::Lexergains a constructor that takes that start position (line, column, offset). This adds to the public API. The existing constructor is unchanged.Evidence
A Go probe run against protowire-go 903b4a7, with
MaxBytesLiteralLength: 191, gives these lines. The new testDatasetReaderPositions.MatchProtowireGoexpects them verbatim:2:3: bytes literal decodes to more than …=191 bytes1:3: …pos=2:1@23,3:1@29; error5:5: …1:1@0,1:1@0;1:3: …pos=1:24@23, cell1:26@25; error1:34: …1:1@0,1:3@2;1:3: …2:5: …2:5: …(matched already: that row starts on line 1)3:4: @dataset row has 1 cells, expected 2 (column count)1:1: …3:7: expected ',' or ')' in @dataset row, got integer1:5: …3002:3: @dataset row has 2 cells, expected 1 (column count)Scan, a column the type lacks3:3: @dataset column "nope": google.protobuf.FieldDescriptorProto has no field "nope"Scan, a cell of the wrong type3:3: column number: expected integer for field "number"1:1: …With the fix stashed, 8 of the 9 cases fail. The exception is "a row's second line", for the reason in the table. With the fix,
ctestpasses 729/729 locally (macOS, Homebrew protobuf).Classification
Caveat
This machine has clang-format 23, while the CI job runs 18. The new lines are clean under 23, and the CI clang-format job will check them under 18.