DatasetReader re-lexes a long token from its start on every read #168

DatasetReader (after #167) finds a row's end with the lexer and resumes at the first token it has not yet seen whole. Every token that is complete is lexed once. A single long token that spans many 4096-byte reads is the exception: it is lexed again from its start on each read, until it ends. A multi-megabyte string or bytes cell, or a long comment between rows, costs time quadratic in its length. 4 MiB is 1,024 reads, and each re-lexes up to 4 MiB.

protowire-go avoids this with mayEndToken (encoding/pxf/dataset_stream.go, #148 there). While a token waits, it does not lex again until the bytes read since could end it: a closing quote not escaped by an odd run of backslashes, a newline, """, */, a byte that ends a run, or a bytes literal reaching MaxBytesLiteralLength. Go's scanned counter and dataset_scan_linear_test.go hold the scan linear.

Before #167 the byte scanner rescanned the whole buffer from its start on every read, so this is not a regression. The quadratic case is narrower than it was.

This is not measured in cpp. The cost is read off the code: ScanRow in src/pxf/dataset_reader.cc lexes from scan_pos_, which stays at a token's start while it is incomplete. ScanHeader does not resume at all: it lexes the buffer from its start on every read. The 64 KiB header cap bounds that.

Done when

• ScanRow and ScanHeader lex a waiting token again only when the new bytes could end it, as Go's mayEndToken decides.
• A test holds the bytes lexed linear in the input, for a long string, a long bytes literal and a long comment, each spread over many reads (Go's dataset_scan_linear_test.go is the model).
• testdata/dataset-stream/go_sweep.tsv still matches.