[bfrancojr]

Fixes #171. Stacked on #172 (#168). #167 has merged. It also carries #170's commit (95ab86f, the #169 timestamp fix, identical to #170's), because one of the live rows, (2026-10-23T00:00:00Z), needs it. Merge order: #172, #170, then this. Rebasing after #170 merges drops the duplicate commit.

What was wrong

On a pipe, a socket or a file still being appended to, Next could hold a row that had already arrived. It waited for 4 KiB more input, or for the stream to end. Two causes:

1. Pull used std::istream::read, which waits for all 4096 bytes.
2. The scan took a token as complete only once four more bytes followed it. A row whose ) was the last byte read waited.

What changes

• Pull takes what the stream has buffered (readsome). It blocks only when nothing is buffered, through peek, for one byte or the end. On an istringstream the reads are still 4096 bytes, so the sweep's byte counts are unchanged.

• A token counts as complete as soon as the bytes the lexer looks at past it have arrived:

  • a one-byte token (punctuation, a newline) needs none;
  • a closed string, bytes literal or block comment needs none, except "", which a third quote would turn into """;
  • any other token needs the byte after it, and the byte after that when the first is ., p, / or the first byte of µ;
  • an error token, or one that starts past ASCII, still waits for four.

  That follows what the lexer reads, path by path. Go's lexer records the same thing as sawEnd.

The issue left the read method as a choice for the PR. I chose readsome plus peek over rdbuf()->sgetn, so the stream's state flags stay right for the caller.

Evidence

• DatasetReaderLive.CompleteRowIsNotHeldBack feeds protowire-go's 16 rows (TestDatasetReader_CompleteRowIsNotHeldBack) one byte per read. It uses a streambuf that records any read past the bytes written, which stands in for a pipe that stays open, without threads or timeouts. Before this change all 16 rows waited. With it, none does.
• The first version of the rule still held 2 of the 16: the triple-quoted string and the escaped string, both closed at the last byte read. Treating closed strings as complete fixed both.
• The 590-case Go sweep from pxf: DatasetReader ends a table only at the end of input or a directive (#164) #167 still matches. That sweep is what would catch a token taken as complete too early, at every read boundary in 14 row shapes.
• DatasetReaderScan.IsLinearOneByteAtATime keeps the scan linear in one-byte reads, which Pull can now return: 4 shapes at 256 KiB. It also passes on the old reader, which never did one-byte reads, so it guards the new path rather than proving this fix.
• ctest: 735/735 locally (macOS). clang-format 18 is clean. clang-tidy 23 reports nothing new on the changed files.
• Throughput on pxf: DatasetReader ends a table only at the end of input or a directive (#164) #167's 300,000-row table: 83–88 ms, against 83–86 ms before (best of 5, local). That's within noise.
• CI on 4095d75 (head, rebased onto main 42d3c27, which has pxf: DatasetReader lexes a long token once, however many reads it spans (#168) #172): all 11 checks pass, including CodeQL, ASan + UBSan, the differential against protowire-go, clang-tidy and Windows. Locally 735/735.