Skip to content

pxf: keep list, block and precision in dataset cells - #52

Merged
trendvidia merged 2 commits into
mainfrom
fix/47-structured-cells
Nov 8, 2026
Merged

trendvidia merged 2 commits into
mainfrom
fix/47-structured-cells

Conversation

@bfrancojr

Copy link
Copy Markdown
Collaborator

Fixes #47, by the option the owner chose there (option 2).

What changes

A row read as cells (DatasetReader.next_or_none, iteration, Result.datasets) now keeps what protowire-go's DatasetRow keeps. Passed to bind_row, it binds as DatasetReader.scan binds it. Before:

  • a list or block cell came back as ("unknown", None), and bind_row refused it;
  • 1.5p80 came back as ("float", "1.5"), and a BigFloat bound without its precision.

The public Cell type gains two kinds, and the precision suffix stays in the raw text:

("list", [("int", "1"), ("string", "s"), ("list", [("int", "2")])])
("block", [
    ("type", "t/x", None, False),                           # @type = "t/x"
    ("field", "k", ("int", "1"), False),                    # k = 1
    ("block", "m", [("field", "n", ("int", "2"), False)], False),  # m { n = 2 }
    ("map", "q", ("int", "3"), True),                       # "q": 3
])
("float", "1.5p80")

An entry is (kind, key, value, quoted), one kind for each of cpp's entry types (Assignment, MapEntry, Block, TypeEntry). quoted keeps a key written as a string literal, which tells the string map key "1" from the integer key 1. The conversion in module.cc is now recursive in both directions. On the way back the suffix is split off at its p. That's unambiguous, because PXF numbers are decimal: cpp's LexNumber has no hex form, and LexPrecision takes p followed by digits.

Breaking for callers: float(cell[1]) now raises on a suffixed number, where before it silently dropped the precision. This is in the changelog.

Evidence

Built from this branch into a fresh venv (protowire-cpp v2.1.0, CPython 3.15, macOS arm64): pytest tests → 207 passed, the 200 from main plus 7 new in tests/test_pxf_bind_row.py:

An unreleased changelog line from #24, "a cell may hold a list or a block (scan only; see #47…)", is updated, since scan is no longer the only way.

A row's cells as Python saw them lost part of the row: a list or block
cell came back as ("unknown", None), and a number's precision suffix was
dropped (1.5p80 -> ("float", "1.5")). So a row read with next_or_none
and passed to bind_row did not bind as scan binds it, and Python's cells
held less than protowire-go's DatasetRow, whose cells are the full parse
tree (ListVal.Elements, BlockVal.Entries, IntVal/FloatVal.Prec).

The owner chose option 2 on #47: lists and blocks become structured
cells, ("list", [cell, ...]) and ("block", [entry, ...]), and the suffix
stays in the raw text. An entry is (kind, key, value, quoted), one per
kind of cpp's EntryPtr: Assignment, MapEntry, Block and TypeEntry, with
the key's quoting kept, since it tells the string map key "1" from the
integer key 1. The conversion both ways is now recursive. The suffix is
split back off at its 'p', which is unambiguous since PXF numbers are
decimal (cpp's lexer: LexNumber has no hex form, LexPrecision takes 'p'
and digits).

Fixes #47
@bfrancojr

Copy link
Copy Markdown
Collaborator Author

CI at 9e34f38: all 18 checks pass (Linux, macOS and Windows on py3.10–3.13, the three cibuildwheel builds, CodeQL).

@trendvidia
trendvidia merged commit 3e59fbb into main Nov 8, 2026
18 checks passed
@trendvidia
trendvidia deleted the fix/47-structured-cells branch November 8, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dataset cells lose list/block values and precision suffixes, so bind_row on them differs from scan

2 participants