fix(cpp): TS_2DIFF float/double maxPointNumber once per page (fixes #910)

[kkzi]

Fix C++ TS_2DIFF float/double encoding to match the Java layout, and make the decoder tolerate all three page layouts. Fixes #910.

Summary

• Encoder: write the maxPointNumber var_uint (fixed value 2) exactly once per page instead of at every segment boundary, matching Java FloatEncoder/DoubleEncoder. The Java readers (e.g. TsFileSketchTool) crash on the old layout when a page's first segment is empty/short — that is fix(cpp): Float/DoubleTS2DIFFEncoder writes maxPointNumber per segment, breaking Java FloatDecoder on multi-segment pages #910. The encoder writes maxPN at page start (first encode after reset), so every non-empty page begins with either a FLAG section or the maxPN prefix — this is the invariant the decoder relies on.
• Decoder: forward-only prefix-aware parsing that accepts legacy raw pages (no prefix, first byte 0x00), the new Java layout (maxPointNumber only on the page's first segment), and the old C++ per-segment format (backward compatible). A prefix-free segment's header is preloaded so decode() never rewinds the stream.
  • Note on approach: this replaces the earlier whole-page scan (scan_java_float_double_page in 1ef5e94). The scan assumed every segment carries a prefix and could not bound segment boundaries on new-format pages (segments 2+ have no prefix and no separator), which made scaled-overflow pages (FLAG + prefix-free continuation) unreadable. The forward-only parser dispatches on the first byte (0x00 / 0x02 / FLAG) with a per-page page_first_segment_ flag, which handles the new layout unambiguously.
• ByteStream: check_space() recomputes the read page from the head instead of blindly following next_ when the cursor is parked at a page boundary (from 1ef5e94). The decoder's probe rewinds (set_read_pos fallback branches) rely on this.
• Tests: new gtest cases assert the once-per-page byte layout for multi-segment pages, scaled-overflow pages (the fix(cpp): Float/DoubleTS2DIFFEncoder writes maxPointNumber per segment, breaking Java FloatDecoder on multi-segment pages #910 crash scenario), reset() page boundaries, and legacy per-segment backward compatibility. Legacy raw batch/scalar/mixed regressions from the earlier review are kept.

Verification

• Full C++ test suite: 763/763 pass.
• Java TsFileSketchTool reads files written by the fixed encoder (previously crashed).
• tsfile_cli round-trips the data.

[ColinLeeo]

Thanks for tracking this down.

The root-cause analysis is clear, and the new implementation correctly handles Java-compatible prefixes, including overflow prefixes and reads spanning multiple segments.

I found one blocking compatibility issue, though: routing FLOAT/DOUBLE batch reads through the scalar decoder regresses legacy raw segments. The scalar prefix detector can misclassify a valid raw header, after which the decoder gets an invalid bit_width_ and spins at end-of-input.

I reproduced this for both FLOAT and DOUBLE by encoding 129 sequential raw bit patterns with IntTS2DIFFEncoder / LongTS2DIFFEncoder, then reading them in small batches through the corresponding floating-point decoder. The PR head hangs, while the parent implementation completes successfully.

Could we preserve the integer batch path for legacy raw segments, or make the prefix detection unambiguous before switching to the scalar path? It would also be good to add legacy raw batch regression tests for both types.

[ColinLeeo]

The overall fix direction looks good, but the legacy raw segment compatibility issue is not fully addressed yet.

[kkzi]

Hi @ColinLeeo, thanks for the thorough review and the reproduction steps — they made this straightforward to chase down. I've pushed 1ef5e94 addressing all three points.

Root cause confirmed. Your repro hangs exactly as described: the per-block heuristic (looks_like_ts2diff_header) only validated a single misaligned block header (wi/bw range check). A legacy raw block with wi = 0 (or bit_width = 0, i.e. any constant-value block) passes that probe with all-zero bytes, gets misread as a maxPointNumber prefix, and the desync cascades into the end-of-input spin.

Fix — unambiguous prefix detection (your option 2). The layout is now decided once per page by scan_java_float_double_page(), which parses the entire remaining stream with the Java segment grammar: [overflow flag][count][bitmaps][mpn] block+, with field-range validation, Σ(wi+1) == bitmap count for overflow segments, and exact whole-stream exhaustion. A page only counts as Java-compatible when the grammar consumes it exactly. A real legacy raw page fails this immediately: after the varint tag eats the leading 0x00, the misaligned write_index probe reads >= 0x100 and is rejected. The detected prefix offsets are recorded and the decoder only consumes prefixes at those offsets, which also fixes value semantics for Java single-segment multi-block pages the per-block heuristic couldn't represent.

Legacy raw batch path preserved (your option 1). Legacy raw pages route through the integer SIMD batch decoder + bit-cast, exactly the parent-commit behavior; Java pages take the segment-aware scalar path.

Regression tests. Added for both FLOAT and DOUBLE:

• ReadBatchFloatLegacyRawSegments / ReadBatchDoubleLegacyRawSegments — 129 values via IntTS2DIFFEncoder/LongTS2DIFFEncoder (128-value constant block + trailing change, hitting both the bit_width = 0 and wi = 0 misclassification patterns), read in small batches of 16
• ReadFloatLegacyRawScalar / ReadDoubleLegacyRawScalar — scalar path across the block boundary
• LegacyRawBatchThenScalarReads — mixed batch/scalar reads on one page

Also hardened read_long() to bail out when the stream is exhausted with bits still owed, so no residual misconfiguration can loop forever.

One incidental fix this surfaced: ByteStream::check_space() skipped a page when set_read_pos() parked the cursor at a page boundary (it blindly followed read_page_->next_, yielding E_OUT_OF_RANGE on the next read). The scan-based detection depends on position restore, so I fixed it to recompute the page from the head.

Full C++ suite (757 tests) passes, and clang-format --dry-run --Werror is clean. Happy to adjust if you'd prefer a different split.

[kkzi]

Thanks for the review. I have pushed a66a679 which fixes the spotless clang-format violations flagged by CI (ts2diff_decoder.h and ts2diff_codec_test.cc).

The CI runs for this new commit are currently waiting for approval (action_required) — could you approve the workflows so they can re-run?

Happy to address any remaining feedback on the legacy-raw compatibility path.