Repository navigation
pxf: bind a dataset row through protowire-cpp's BindRow - #48
Merged
Merged
Conversation
bind_row wrote each cell back as PXF text and decoded the body with unmarshal. That skipped (pxf.default) and (pxf.required), refused list and block cells and dotted columns, and gave errors at the body's positions. scan did the same on cells already turned into Python tuples. bind_row now rebuilds the row's AST cells in the bridge and calls pxf::BindRow. scan keeps the row native and binds it there, as DatasetReader::Scan does: under the reader's limits but max_message_size, with the schema checked once per type per reader. On all 43 fixtures in the spec's testdata/dataset/, scan gives protowire-go v1.17.0's message or error, text and position included (testdata/dataset-scan.go.txt). As in Go, decided on #24: - a row binds into what the message holds, which is no longer cleared; - skip_validate defaults to False for both, as Go and unmarshal check. Both take discard_unknown. Cells read with next_or_none still lose list and block values and precision suffixes; that is #47. Fixes #24
Collaborator
Author
|
CI on cbc3f55: all 12 test legs (Linux, macOS, Windows × py3.10–3.13), the 3 cibuildwheel smoke builds and CodeQL (cpp, python) pass. Test count read from the ubuntu py3.10 and windows py3.13 logs: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pxf.bind_rowandDatasetReader.scannow bind a row through protowire-cpp'sBindRow, so a row binds as protowire-go binds it. Fixes #24.What changes for callers
(pxf.default). An absent(pxf.required)field is an error. Before, both were silently skipped.scanerror carries the row's line and column in the input and names the column, e.g.8:1: column int32_field: null is not allowed ….nullcell. The docs say to bind each row into a new or cleared message.skip_validatenow defaults to False, as in Go andunmarshal.scanchecks once per message type;bind_rowchecks on every call.discard_unknown.How
bind_rowturns the Python cells back into cpp AST values in the bridge and callspxf::BindRow.scankeeps the row in C++ and binds it there, asDatasetReader::Scandoes. That means the reader's limits exceptmax_message_size, and the schema checked once per type. Its descriptor set is cached per reader, rather than rebuilt per row as before.unmarshal, is gone.Evidence
All 43 spec dataset fixtures (
testdata/dataset/at protowire dd8e221), streamed throughscanwith a fresh message per row. Output compared with protowire-go v1.17.0'sDatasetReader.Scan, each message as its deterministic encoding in hex:pxf.DatasetReader.scan:prefix).unknown cell kind 'unknown'orunknown field "nested_field.name", or bind differently.Go's output is checked in as
testdata/dataset-scan.go.txt.test_every_spec_dataset_fixture_streams_as_gocompares against it when the spec repo is checked out next to this one. CI has no spec checkout, so that test skips there. The tests that run in CI copy the fixtures they need verbatim.#24's done-when, measured against Go:
0a01611201781a017a(f "a", g "x", h "z"). Python gives the same.3:1: required field "f" is absent. Pythonscangives the same.bind_rowgives1:1: …, because a row passed as cells has no position in its input.Reused message, Go v1.17.0: after row 2 of
("a", "x", 1)/("b", null, ), the same message holdsint32_field: 1andnullable_string: "x". Pinned intest_a_row_binds_into_what_the_message_holds.Tests, macOS, Python 3.15, local build against cpp v2.1.0:
200 passed, the golden-file test included.14 failed, 186 passed. The 14 are exactly the new or changed tests.HARDENING gate (
REQUIRE_PORTS=python,cpp):Ran 136 (port, corpus) pairs across 2 ports. ✓Not covered here
next_or_noneorunmarshal_fullstill lose list and block values (they come back as("unknown", None)) and precision suffixes (1.5p80→("float", "1.5")). Sobind_rowon such a row differs fromscan. Fixing it changes the publicCelltype, so it's filed with options, not fixed here.