[bfrancojr]

Fixes #168. Built on #167 (#164), which brought in the lexer-based scan this changes. #167 has merged, and this PR is rebased onto main (531c586).

What was wrong

A token still open at the end of a 4 KiB read was lexed again from its start on every read. That covers an unclosed string, a bytes literal, a comment, or one long number or identifier. A token of n bytes cost about n²/8192 byte visits: 136 MB of lexing for 1 MiB, 540 MB for 2 MiB. The header scan lexed the whole buffer from the start on every read.

What changes

The reader now does what protowire-go's reader does (mayEndToken). The scan records the open token, and lexes it again only when the bytes read since could end it:

• a closing quote not escaped by an odd run of backslashes;
• a newline;
• """ or */;
• a byte that ends a run, for a timestamp;
• a byte that cannot continue an identifier or number;
• a bytes literal reaching max_bytes_literal_length.

A token under 8 bytes is always lexed again. The rule only decides when the lexer looks again, never what the token is. The header scan now keeps its place between reads too. An @proto body is matched by BraceMatch, a resumable form of FindMatchingBrace, which now wraps it.

Evidence

Bytes lexed, counted through a test peer (DatasetReaderTestPeer), for an unclosed row after the header, read 4 KiB at a time:

Shape	1 MiB, with this PR	1 MiB, waiting turned off	2 MiB, waiting turned off
unterminated string	1,052,659	135,786,499	540,008,451
one long number	1,052,658	135,786,498	540,008,450
line comment	1,052,660	135,785,990	540,007,430
one long bytes literal	1,052,660	135,786,500	540,008,452
short tokens (control)	1,048,834	1,048,834	2,097,666

The off figures fit n²/8192, which is 134 M for 1 MiB, and they grow 4× for 2× the input. All 10 shapes behave the same way: Go's 9, plus a long bytes literal. For a header near 64 KiB with one long string, 65,555 bytes are lexed against 552,853 with waiting off.

• DatasetReaderScan.IsLinear and HeaderIsLinear allow twice the input. With the waiting rule turned off (a local patch, not committed), 18 of the 20 row cases fail and so does the long-string header case. The two that pass are the short-tokens control.
• The 590-case Go sweep from pxf: DatasetReader ends a table only at the end of input or a directive (#164) #167 still matches.
• ctest: 732/732 locally (macOS). clang-format 18 is clean. clang-tidy 23 on the changed sources and headers reports nothing new.
• Throughput on pxf: DatasetReader ends a table only at the end of input or a directive (#164) #167's 300,000-row table: 83–86 ms, against 81–85 ms before this PR (best of 5, local). That's within noise.
• CI on 38a9ca2 (head, rebased onto main 222ef6c, which has pxf: a timestamp ends at ')', as protowire-go's lexer ends it (#169) #170): all 11 checks pass, including CodeQL, ASan + UBSan, the differential against protowire-go, clang-tidy and Windows. Locally 733/733.

Not in this PR

Go's third test in the same file, CompleteRowIsNotHeldBack, needs a live stream, and cpp's Pull uses std::istream::read, which waits for a full 4 KiB. That's #171. Go's one-byte-reader variants of these tests have no cpp equivalent until #171, since every read is 4 KiB or the rest of the input.