Repository navigation
pxf: keep list, block and precision in dataset cells - #52
Merged
Merged
Conversation
A row's cells as Python saw them lost part of the row: a list or block
cell came back as ("unknown", None), and a number's precision suffix was
dropped (1.5p80 -> ("float", "1.5")). So a row read with next_or_none
and passed to bind_row did not bind as scan binds it, and Python's cells
held less than protowire-go's DatasetRow, whose cells are the full parse
tree (ListVal.Elements, BlockVal.Entries, IntVal/FloatVal.Prec).
The owner chose option 2 on #47: lists and blocks become structured
cells, ("list", [cell, ...]) and ("block", [entry, ...]), and the suffix
stays in the raw text. An entry is (kind, key, value, quoted), one per
kind of cpp's EntryPtr: Assignment, MapEntry, Block and TypeEntry, with
the key's quoting kept, since it tells the string map key "1" from the
integer key 1. The conversion both ways is now recursive. The suffix is
split back off at its 'p', which is unambiguous since PXF numbers are
decimal (cpp's lexer: LexNumber has no hex form, LexPrecision takes 'p'
and digits).
Fixes #47
Collaborator
Author
|
CI at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #47, by the option the owner chose there (option 2).
What changes
A row read as cells (
DatasetReader.next_or_none, iteration,Result.datasets) now keeps what protowire-go'sDatasetRowkeeps. Passed tobind_row, it binds asDatasetReader.scanbinds it. Before:("unknown", None), andbind_rowrefused it;1.5p80came back as("float", "1.5"), and a BigFloat bound without its precision.The public
Celltype gains two kinds, and the precision suffix stays in the raw text:An entry is
(kind, key, value, quoted), one kind for each of cpp's entry types (Assignment,MapEntry,Block,TypeEntry).quotedkeeps a key written as a string literal, which tells the string map key"1"from the integer key1. The conversion inmodule.ccis now recursive in both directions. On the way back the suffix is split off at itsp. That's unambiguous, because PXF numbers are decimal: cpp'sLexNumberhas no hex form, andLexPrecisiontakespfollowed by digits.Breaking for callers:
float(cell[1])now raises on a suffixed number, where before it silently dropped the precision. This is in the changelog.Evidence
Built from this branch into a fresh venv (protowire-cpp v2.1.0, CPython 3.15, macOS arm64):
pytest tests→ 207 passed, the 200 from main plus 7 new intests/test_pxf_bind_row.py:test_a_row_read_as_cells_binds_as_scan_binds_it[list-cell, block-cell, nested-cells, cell-framing]: each row from iteration, bound withbind_row, gives the bytes protowire-go v1.17.0 gives for that fixture (the_ACCEPTtable from pxf.bind_row promises (pxf.default) and (pxf.required) but applies neither #24). These are the four fixtures Dataset cells lose list/block values and precision suffixes, so bind_row on them differs from scan #47's done-when names.nested-cellscovers map entries and nested blocks.test_a_precision_suffix_survives_the_cell:(1.5p80)reads as("float", "1.5p80"), binds withprec == 80, and gives the same bytes asscan.test_list_and_block_cells_keep_their_parts: the shape above, all four entry kinds plus a quoted key and a nested list.test_a_list_element_cannot_be_absent:("list", [None])is aValueError.An unreleased changelog line from #24, "a cell may hold a list or a block (
scanonly; see #47…)", is updated, sincescanis no longer the only way.