Skip to content

pxf: bind a dataset row through protowire-cpp's BindRow - #48

Merged
trendvidia merged 1 commit into
mainfrom
bind-row-via-cpp
Nov 7, 2026
Merged

trendvidia merged 1 commit into
mainfrom
bind-row-via-cpp

Conversation

@bfrancojr

Copy link
Copy Markdown
Collaborator

pxf.bind_row and DatasetReader.scan now bind a row through protowire-cpp's BindRow, so a row binds as protowire-go binds it. Fixes #24.

What changes for callers

  • Defaults and required fields apply. An absent cell takes the field's (pxf.default). An absent (pxf.required) field is an error. Before, both were silently skipped.
  • Value cells and column paths work. A cell may hold a list or a block, and a column may be a dotted path into a sub-message. Before, both were refused.
  • Errors point at the row. A scan error carries the row's line and column in the input and names the column, e.g. 8:1: column int32_field: null is not allowed ….
  • A row binds into what the message already holds (your call on pxf.bind_row promises (pxf.default) and (pxf.required) but applies neither #24). The message is no longer cleared first, as in Go. If one message is reused across rows, a field set by an earlier row keeps its value under an empty or null cell. The docs say to bind each row into a new or cleared message.
  • The schema is checked by default (your call on pxf.bind_row promises (pxf.default) and (pxf.required) but applies neither #24). skip_validate now defaults to False, as in Go and unmarshal. scan checks once per message type; bind_row checks on every call.
  • Both take discard_unknown.

How

  • bind_row turns the Python cells back into cpp AST values in the bridge and calls pxf::BindRow.
  • scan keeps the row in C++ and binds it there, as DatasetReader::Scan does. That means the reader's limits except max_message_size, and the schema checked once per type. Its descriptor set is cached per reader, rather than rebuilt per row as before.
  • The old path, which wrote cells back out as PXF text and decoded with unmarshal, is gone.

Evidence

All 43 spec dataset fixtures (testdata/dataset/ at protowire dd8e221), streamed through scan with a fresh message per row. Output compared with protowire-go v1.17.0's DatasetReader.Scan, each message as its deterministic encoding in hex:

  • This branch: identical to Go, 32 bound rows plus every error's text and position (after removing this port's pxf.DatasetReader.scan: prefix).
  • main: most fixtures fail with unknown cell kind 'unknown' or unknown field "nested_field.name", or bind differently.

Go's output is checked in as testdata/dataset-scan.go.txt. test_every_spec_dataset_fixture_streams_as_go compares against it when the spec repo is checked out next to this one. CI has no spec checkout, so that test skips there. The tests that run in CI copy the fixtures they need verbatim.

#24's done-when, measured against Go:

  • Absent default: Go row 1 gives 0a01611201781a017a (f "a", g "x", h "z"). Python gives the same.
  • Absent required: Go gives 3:1: required field "f" is absent. Python scan gives the same. bind_row gives 1:1: …, because a row passed as cells has no position in its input.

Reused message, Go v1.17.0: after row 2 of ("a", "x", 1) / ("b", null, ), the same message holds int32_field: 1 and nullable_string: "x". Pinned in test_a_row_binds_into_what_the_message_holds.

Tests, macOS, Python 3.15, local build against cpp v2.1.0:

  • Branch: 200 passed, the golden-file test included.
  • The same tests against a build of main: 14 failed, 186 passed. The 14 are exactly the new or changed tests.

HARDENING gate (REQUIRE_PORTS=python,cpp): Ran 136 (port, corpus) pairs across 2 ports. ✓

Not covered here

bind_row wrote each cell back as PXF text and decoded the body with
unmarshal. That skipped (pxf.default) and (pxf.required), refused list
and block cells and dotted columns, and gave errors at the body's
positions. scan did the same on cells already turned into Python tuples.

bind_row now rebuilds the row's AST cells in the bridge and calls
pxf::BindRow. scan keeps the row native and binds it there, as
DatasetReader::Scan does: under the reader's limits but max_message_size,
with the schema checked once per type per reader. On all 43 fixtures in
the spec's testdata/dataset/, scan gives protowire-go v1.17.0's message
or error, text and position included (testdata/dataset-scan.go.txt).

As in Go, decided on #24:
- a row binds into what the message holds, which is no longer cleared;
- skip_validate defaults to False for both, as Go and unmarshal check.

Both take discard_unknown. Cells read with next_or_none still lose list
and block values and precision suffixes; that is #47.

Fixes #24
@bfrancojr

Copy link
Copy Markdown
Collaborator Author

CI on cbc3f55: all 12 test legs (Linux, macOS, Windows × py3.10–3.13), the 3 cibuildwheel smoke builds and CodeQL (cpp, python) pass. Test count read from the ubuntu py3.10 and windows py3.13 logs: 194 passed, 6 skipped, which is 200, the local count. The 6 skips are the 5 SBE view tests and test_every_spec_dataset_fixture_streams_as_go, all of which need a spec checkout that CI doesn't have. The 43-fixture comparison ran locally only (200 passed).

@trendvidia
trendvidia merged commit 08fc470 into main Nov 7, 2026
18 checks passed
@trendvidia
trendvidia deleted the bind-row-via-cpp branch November 7, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pxf.bind_row promises (pxf.default) and (pxf.required) but applies neither

2 participants