Skip to content

Run the spec's whole per-port corpus through both tiers, from one copy CI checks - #216

Open
bfrancojr wants to merge 3 commits into
mainfrom
lite/corpus-parity
Open

bfrancojr wants to merge 3 commits into
mainfrom
lite/corpus-parity

Conversation

@bfrancojr

@bfrancojr bfrancojr commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Both Java tiers now run every test file in the spec's per-port corpus, from one copy that CI keeps equal to the spec. Before, the lite tier ran hand-typed copies of 2 of the 12 folders and none of the rest, and nothing noticed when the spec changed a file.

Fixes #211.

What changes

  • One copy. testdata/<dir> is protowire's testdata/<dir> byte for byte, schemas included: annotations, any, bigfloat, blocks, dataset, duplicates, duration, keyed, map-keys, nulls, repeated, strings, plus test.proto. It replaces the copies under pxf/src/test/resources and pxf/src/test/proto, which were taken from spec 234cf29 and had fallen behind (they lacked protowire#569's three required-field files).
  • Full tier. It compiles those schemas and runs their files. SpecCorpusTest adds the three folders it never ran (annotations, duration, repeated). It also fails if a copied folder has no test.
  • Lite tier. lite-fixtures generates lite classes from the same schemas. The java_package option that the lite plugin requires is added at build time, so no edited copy is checked in. LiteCorpusTest runs every file through LiteWireWriter, and through LiteWireReader where the corpus gives expected bytes or text.
  • Known gaps. A file a tier doesn't handle yet is listed against its open issue. A listed file that starts passing fails the test, so the list only shrinks.
  • CI. scripts/check-spec-corpus.sh runs in the conformance job against spec main. It fails when a copy differs from the spec, or when the spec has a corpus folder that is neither copied nor listed with what covers it instead.

What the first run found

Lite gaps, each filed and listed:

Both tiers: the 3 required-field null files from protowire#569 (#189, which is the next PR).

Evidence

  • ./gradlew build --rerun-tasks with the CI plugin pin (-Dorg.gradle.project.pxfJavaMeta.repo=/nonexistent): BUILD SUCCESSFUL, 947 tests, 0 failures, every result file freshly written. That count reconciles with main's: +7 SpecCorpusTest, +1 gap-existence check, +123 LiteCorpusTest cases, −19 hand-typed LiteTablesTest rows.
  • LiteCorpusTest runs 123 cases: 110 files plus 13 named tests. I counted the files by hand per folder to reconcile.
  • The gap check fires both ways. Before listing, each gap failed with the messages quoted in pxf lite: an Any block's @type entry is refused, so documents Go reads fail on lite #213–pxf lite: three errors (bool map key, repeated oneof member, repeated int map key) don't say what Go says #215. With a passing file (blocks/space-in-block.pxf) listed as a gap, the test fails with meets its verdict now: drop it from KNOWN_GAPS (#0).
  • scripts/check-spec-corpus.sh against spec main 8e08cd9: testdata/: 12 directories match the spec's. Against spec 234cf29, with a made-up folder added, it names annotations, dataset, keyed and nulls as differing, and flags newcorpus as neither copied nor listed.
  • The spec's cross_envelope_check.sh at 8e08cd9, against this branch, with both lite legs on (WITH_JAVA_LITE=1, WITH_JAVA_PXF_LITE=1), exited 0. All 6 ports write identical envelope bytes, and every annotation and wire-vector row is ok for java-lite and java-pxf-lite. Rust, Swift, Dart and C# were off.
  • Not run locally: the text, write and security gates. They drive check-decode and check-decode-android, which this PR doesn't touch. This PR's conformance job runs them.

Also fixed

  • pxf-android's KeyedLiteTest reads testdata/keyed/, not the deleted pxf/src/test/resources/keyed/.
  • The struct-binding table step in CI no longer skips when spec main has no values.tsv. protowire#518 merged, and the table matches (cmp clean).

testdata/<dir> is protowire's testdata/<dir> byte for byte, schemas
included, for annotations, any, bigfloat, blocks, dataset, duplicates,
duration, keyed, map-keys, nulls, repeated and strings, plus test.proto.
It replaces the copies under pxf/src/test/resources/{corpus,keyed,map-keys}
and pxf/src/test/proto, taken from spec 234cf29 and since overtaken: nulls
and dataset lacked the three (pxf.required) fixtures of protowire#569 and
the messages they bind.

The full tier compiles the schemas as test protos (all but the v1.2-only
settings_annotated.proto and the editions schema, which protoc 3.25.5
reads only behind a flag) and reads the fixtures by path: the corpus
directory names are also the generated packages' class directories, so
a classpath directory lookup finds those first.

SpecCorpusTest drives the three directories nothing ran here (annotations,
duration, repeated) and fails when a vendored directory has no driver.
A fixture the tier does not meet is listed in CorpusFixtures.KNOWN_GAPS
against its issue, and a listed fixture that meets its verdict fails, so
the list only shrinks. Today it lists the three #189 fixtures.
lite-fixtures now generates its lite classes and companions from
testdata/'s schemas. protoc-gen-pxf-java-meta requires a java_package
and the spec's schemas carry none, so a Sync task adds one (the proto
package, protoc's own default) and java_multiple_files on the way to
protoc; nothing else in a file changes. The hand copies of the nulls,
keyed and settings schemas go. settings_annotated.proto stays: it needs
a v1.2 compiler, and src/main/proto spells it for stock protoc.

LiteCorpusTest runs every fixture of all twelve directories through
Parser + LiteWireWriter and, where the corpus gives bytes or text, back
through LiteWireReader: verdicts by file name, bigfloat's encode and
decode cases, the annotations golden, duration values and the negative
round trip, concatenated lists, the map-key fmt pairs, the @dataset
parse verdicts, and the error texts the full tier pins. It replaces
LiteTablesTest's hand-typed nulls rows, one of which (required-optional)
the spec has since made a reject.

The first run found lite gaps, each listed in KNOWN_GAPS against its
issue: #213 (an Any block's @type is refused), #214 (a scalar or an
assigned block on a repeated field is accepted), #215 (three error
texts), #191 (three rejections are IllegalArgumentException) and #189.
@dataset rows are not bound on lite at all (#212), so only the parse
verdicts run.
scripts/check-spec-corpus.sh, run by the conformance job against
protowire main, fails when a directory under testdata/ or test.proto
differs from the spec's, and when the spec has a testdata directory
that is neither copied here nor listed with what covers it instead
(the gates, or cmd/pxf tooling neither tier has). So a fixture the spec
adds reaches both tiers' corpus tests, which then fail until the tier
meets it or lists it as a known gap.

The struct-binding table step loses its guard for a spec main with no
values.tsv: protowire#518 merged, and the table matches.
@bfrancojr

Copy link
Copy Markdown
Collaborator Author

CI on head a314fb7 (run 42518398723): Build + Test and Conformance passed.

  • The new step "Spec corpus copy matches the spec's" ran against spec main and printed testdata/: 12 directories match the spec's.
  • The build job's log has 123 LiteCorpusTest > ... PASSED lines and 7 SpecCorpusTest > ... PASSED lines, which are the counts the PR body gives.
  • The text, write and security gates passed in the conformance job, with java-lite required.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pxf lite: most of the spec corpus never runs through the lite tier, and nothing checks the copies against the spec

1 participant