[bfrancojr]

Fixes #164. Built on #165 (#161, stream positions), whose positions the errors here depend on. #165 has merged, and this PR is rebased onto main (1673cd9).

What was wrong

DatasetReader ended a table cleanly at a row whose ) never came, at an unterminated string, or at anything else where a row could begin. An interrupted append read as a shorter, complete dataset, and every row after a broken one was dropped. Parse refused the same inputs.

What changes

The reader now finds each row's end with the lexer Parse uses, as protowire-go's does. Before, it used its own byte scanner. A row ends at its ), and the table ends at a directive or at the end of the input. Anything else is an error, and it is the error Parse gives for the same input, at the same position. Also as Go does:

• The bytes held while looking for a row's end are capped at max_message_size (dataset input holds N bytes without a row boundary, MaxMessageSize=M). Before, a row that never closed was buffered whole.
• Scan binds a row under the default max_message_size. The body it binds repeats the column names, so it is wider than the row.

Item 3 of the issue gave an example in which Go accepts at max_message_size = 56. Measured, Go refuses that document: its first 4096-byte read already holds 99 bytes. The difference is real, but it shows only when the row arrives in a later read. The sweep has a case built that way (scan body wider than the cap, cap 4500): Go binds it, and the old cpp reader refused it.

Also fixed

• A broken header is refused with Parse's error. Before, Create said no @dataset directive in stream for a header cut short, and gave the byte scanner's own text for an unterminated string. The cause is the same as the row bug, so the header scan was ported from Go as well. It steps over @proto bodies. Measured against Go: 6 of 12 broken headers differed before, and all now match.
• Streaming is faster: a 300,000-row table (4 cells per row) streams in 81–85 ms, against 96–99 ms before, best of 5, locally on macOS. The row fast path parses on the buffer and stops at the ). It drops the comments before the ( instead of copying them.
• One clang-tidy 23 finding on an unchanged line (opts could be moved in Create).

The header fix is in the same commit as the rows: the two share the scan code in dataset_reader.cc.

How it decides a token is complete

cpp's lexer has no sawEnd flag. Go's has one, which says whether the lexer looked past the end of the buffer. Instead, a token counts as complete once the source has ended or four bytes follow it. The farthest the lexer looks past a token is one byte (1.5, 1p8, µs, a / in a timestamp), so later bytes cannot change it. The sweep places the 4096-byte read boundary at every offset in and around 14 row shapes, under three caps. Every one of those cases matches Go.

Evidence

• testdata/dataset-stream/go_sweep.tsv holds protowire-go v1.16.1's (903b4a7) result on 590 inputs, written by tools/dataset_stream_sweep, whose header says how to regenerate it. Test DatasetReaderEnds.MatchProtowireGo replays it. cpp matches all 590 (230 of them errors).
• The same test run against the old reader (src/include stashed, before the header cases were added) failed 219 of the 576 cases it had then. Those included the three fixtures, the Scan case and the cap.
• PxfCorpus.Dataset now reads every dataset fixture through the reader alone. Before, the harness rescanned the tail itself to get the verdict right. It also requires the streamed error text to equal Parse's. All 43 fixtures pass.
• ctest: 730/730 locally (macOS, AppleClang). clang-format 18.1.8 is clean. clang-tidy 23 on the changed sources reports nothing new.
• CI on 50953c5 (head): all 11 checks pass. That covers ASan + UBSan, the differential against protowire-go, clang-tidy, Coverage, CodeQL, gcc-13, clang-18, macOS and Windows (18 min). The run on the rebased a0af71d failed only clang-tidy 18, on performance-enum-size for RowScan. 50953c5 fixes that.

Not done here

• DatasetReader::Scan honours options protowire-go's Scan drops (skip_validate, skip_post_decode, discard_unknown, on_secret_field, validator) #166: Scan passes through options that Go's Scan drops (skip_validate, skip_post_decode, discard_unknown, on_secret_field, validator). cpp documents skip_validate, so the fix needs your decision.
• DatasetReader re-lexes a long token from its start on every read #168: one long token spread over many reads (a multi-megabyte string, say) is lexed again from its start on each read. Go avoids that with mayEndToken. Every complete token before it is not lexed again. The old byte scanner rescanned the whole buffer on every read, so this is no worse than before.

trendvidia/protowire-python#45 is blocked on this. I'll comment there when it merges.