Skip to content

ci(release): a release could ship with fewer conformance cases passing - #212

Open
jamesconwaydev wants to merge 3 commits into
mainfrom
release-gates
Open

jamesconwaydev wants to merge 3 commits into
mainfrom
release-gates

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Fixes #202.

Problem. A release was tagged whenever main was green, and green only means the tests passed. A case that moves under a skip, or out of the loop, drops a conformance count and still passes. The final boss ran only as a 300-page smoke test, and fuzzing ran nightly with nobody required to look at it before a tag.

Change.

  • release.yml runs three gates beside the tag check. build (and so publish, crates-io, and the Python, npm and C workflows, which wait for that GitHub release) needs a gates job, which passes only when all three did:
    • conformance: cargo test --workspace --no-fail-fast with CONFORMANCE_DIR, then scripts/conformance-gate.sh conformance/baseline.json. The release fails if any suite passes fewer cases or fails more, or if a suite in the baseline wrote no counts. The table goes into the run's summary.
    • boss: cargo run --release --locked --manifest-path bench/Cargo.toml -- boss --gate, a new mode that runs all 1,000 pages (giants up to 30 MB included) once through Parsec and html5ever. It fails on a crash, a page over 10 s, a refused page, a page without a document, or any page where the two element counts differ. It takes about a minute on the runner. Locally all 1,000 pages give 0 differences.
    • fuzz: every cargo-fuzz target, listed from crates/*/fuzz/Cargo.toml by cargo metadata, gets five minutes on nightly in its own matrix job. A finding fails the release and is uploaded. A new target, such as parsec-js's, is picked up without a change to the workflow.
  • conformance/baseline.json: each suite's passed, failed and skipped, from the conformance artifact of main's CI run at 7006d31 (32 suites; macOS gives the same counts). test262 and xmlconf aren't on main yet, so they're absent. The gate lists a suite with no entry as "new" and lets it pass, and the release pull request adds it.
  • ci.yml: a new conformance job runs the same comparison on every pull request from the test job's artifact. It isn't a required check, but a drop shows red on the pull request that made it. The test job now writes its counts to target/conformance, since conformance/ holds the baseline. The lint job runs scripts/conformance-gate.test.sh.
  • bench/Cargo.lock still named the crates at 0.1.0 after the 0.2.0 bump, which --locked refuses. It's at 0.2.0 now.
  • CONTRIBUTING.md: how to run the comparison locally, and that the baseline changes only in a release pull request.

Proof. release run 44960154608 is a manual run of release.yml on the throwaway branch release-gates-proof, which skips one passing wpt-url-parsing case. cargo test stays green, and the gate turns the release red:

| wpt-url-parsing | 903 | 0 | 0 | **worse**: baseline 904 passed, 0 failed |
A conformance suite is worse than conformance/baseline.json; see the table above

gates, build, publish and crates-io were skipped. On this branch, release run 44959465290 passes every gate: conformance in 4 min, boss in 1 min, both fuzz targets in 6.5 min, all at once. The same drop run locally:

$ scripts/conformance-gate.sh conformance/baseline.json drop/   # drop/ = main's counts, wpt-url-parsing passed - 1
| wpt-url-parsing | 903 | 0 | 0 | **worse**: baseline 904 passed, 0 failed |
exit 1

Wiki. I haven't pushed the wiki. This patch to Releasing.md adds a "The gates" section, with how the baseline moves, how test262 and xmlconf join it, and speed as a manual gate (bench at load under 3 against the previous tag, with any slowdown over 5% explained in the release notes). It also adds the baseline update to step 2 and "A gate fails" under "If something fails":

Releasing.md patch
diff --git a/Releasing.md b/Releasing.md
index a165721..941e2d3 100644
--- a/Releasing.md
+++ b/Releasing.md
@@ -1,6 +1,6 @@
 This is how a Parsec release goes out, from an empty registry to `cargo add parsec-html`. Most of it happens once, before 0.1.0; after that, a release is a version bump, a changelog entry and a signed tag.
 
-On this page: [What gets published](#what-gets-published) · [How the workflows fit](#how-the-workflows-fit) · [Before the first release](#before-the-first-release) · [Every release](#every-release) · [Check what went out](#check-what-went-out) · [After the release](#after-the-release) · [If something fails](#if-something-fails)
+On this page: [What gets published](#what-gets-published) · [How the workflows fit](#how-the-workflows-fit) · [The gates](#the-gates) · [Before the first release](#before-the-first-release) · [Every release](#every-release) · [Check what went out](#check-what-went-out) · [After the release](#after-the-release) · [If something fails](#if-something-fails)
 
 Every command runs from your home folder as written. Paths are absolute and `gh` gets `--repo`, so nothing depends on where your shell is.
 
@@ -23,12 +23,51 @@ Why six crates and not one? The modules all depend on the parser, so the parser
 
 Pushing a signed `v*` tag starts four workflows at once:
 
-- **`release.yml`** checks the tag's signature against `.github/allowed_signers` on `main`, and that `Cargo.toml`'s version is the tag's. It packages all six crates twice from fresh checkouts, fails if the two differ by a byte, attests them and makes the GitHub release. Then, if `CRATES_PUBLISH` is `true`, its `crates-io` job publishes them in dependency order and checks each one's checksum on crates.io against the attested package.
+- **`release.yml`** checks the tag's signature against `.github/allowed_signers` on `main`, and that `Cargo.toml`'s version is the tag's. It runs [the gates](#the-gates), and nothing is built until they pass. It packages all six crates twice from fresh checkouts, fails if the two differ by a byte, attests them and makes the GitHub release. Then, if `CRATES_PUBLISH` is `true`, its `crates-io` job publishes them in dependency order and checks each one's checksum on crates.io against the attested package.
 - **`release-python.yml`** builds the wheels and the sdist, waits for that GitHub release, then publishes to PyPI if `PYPI_PUBLISH` is `true`.
 - **`release-npm.yml`** builds and tests the package, waits for the release, then publishes to npm if `NPM_PUBLISH` is `true`.
 - **`release-capi.yml`** builds the C packages, waits for the release, then attests them and attaches them to it.
 
-No release, no publishing: a tag with a bad signature or the wrong version stops at the first job, and the other three time out waiting after 30 minutes without publishing anything. All three publish jobs run in the `release` environment and use trusted publishing, so no registry token is stored in GitHub. Each one skips a version that's already up, so a failed run can be run again.
+No release, no publishing: a tag with a bad signature, the wrong version or a failed gate stops before the build, and the other three time out waiting after 30 minutes without publishing anything. All three publish jobs run in the `release` environment and use trusted publishing, so no registry token is stored in GitHub. Each one skips a version that's already up, so a failed run can be run again.
+
+## The gates
+
+`release.yml` runs three gates beside the tag check, and its `build` job waits for its `gates` job, which passes only when all three did. A manual run of `release.yml` runs them too.
+
+| Job | Fails the release when |
+|---|---|
+| `conformance` | Any suite in `conformance/baseline.json` passes fewer cases or fails more than the last release did, or writes no counts at all. `cargo test --workspace` failing fails it too. |
+| `boss` | The final boss (`-- boss --gate`: all 1,000 pages, once, through Parsec and html5ever) crashes, passes 10 s on a page, leaves a page without a document, or builds a different number of elements in Parsec and html5ever on any page. |
+| `fuzz` | Five minutes of `cargo fuzz run` on any target in `crates/*/fuzz` finds a crash. Every target runs at once, one job each; whatever it found is kept as the job's artifact. |
+
+The same comparison against the baseline runs on every pull request as CI's `conformance` job. It isn't a required check, but a drop shows red on the pull request that made it, not on release day. To run it on your machine:
+
+```sh
+cd ~/Documents/GitHub/parsec
+CONFORMANCE_DIR=target/conformance cargo test --workspace
+scripts/conformance-gate.sh conformance/baseline.json target/conformance
+```
+
+### The baseline
+
+`conformance/baseline.json` is each suite's `passed`, `failed` and `skipped` at the last release. It changes only in a release pull request (step 2 of [Every release](#every-release)), never to make another pull request pass. A pull request that trades a case away says so, and the release pull request takes the new counts in on purpose.
+
+A suite that's new since the last release has counts but no entry yet. The gate lists it as new and doesn't hold it to anything; the release pull request adds it, and from then on it's gated like the rest. test262 (from `parsec-js`) and the W3C XML suite (from `parsec-xml`) come in this way when their pull requests land.
+
+### Speed, by hand
+
+CI runners are too noisy to time on, so speed isn't a gate in the workflow. Before tagging, run the benchmark against the previous tag on a quiet machine, with the one-minute load average under 3:
+
+```sh
+uptime    # the first of the three load averages: under 3
+git -C ~/Documents/GitHub/parsec worktree add "$HOME/parsec-prev" v0.1.0    # the previous tag
+cargo run --release --manifest-path "$HOME/parsec-prev/bench/Cargo.toml" > "$HOME/bench-prev.txt"
+cargo run --release --manifest-path ~/Documents/GitHub/parsec/bench/Cargo.toml > "$HOME/bench-now.txt"
+diff "$HOME/bench-prev.txt" "$HOME/bench-now.txt"
+git -C ~/Documents/GitHub/parsec worktree remove "$HOME/parsec-prev"
+```
+
+Any page or table that's more than 5% slower than at the previous tag is explained in the release notes: what got slower, and why it's worth it. If you can't explain it, it's a bug to fix before the release.
 
 ## Before the first release
 
@@ -166,10 +205,23 @@ gh issue view 32 --repo parsec-html/parsec
 gh run list --repo parsec-html/parsec --branch main --workflow ci.yml --limit 1

-### 2. Set the version and the changelog
+### 2. Set the version, the changelog and the baseline

For 0.1.0 this is done: every version is already 0.1.0. For the next one, change version in ~/Documents/GitHub/parsec/Cargo.toml ([workspace.package]), the version = "…" requirements in crates/*/Cargo.toml, bindings/python/Cargo.toml, bindings/wasm/Cargo.toml, bindings/wasm/npm/package.json and the PARSEC_VERSION lines in crates/parsec-capi/cbindgen.toml (then run the cbindgen command at its top, so include/parsec.h follows; a test fails if they disagree), and add the release to CHANGELOG.md, in a pull request like any other.

+The same pull request moves conformance/baseline.json to main's counts, from the conformance artifact of main's latest CI run:
+
+```sh
+run=$(gh run list --repo parsec-html/parsec --branch main --workflow ci.yml --status success --limit 1 --json databaseId -q '.[0].databaseId')
+gh run download "$run" --repo parsec-html/parsec -n conformance -D "$HOME/parsec-conformance"
+jq -s 'map({key: .suite, value: {passed, failed, skipped}}) | sort_by(.key) | from_entries' "$HOME"/parsec-conformance/*.json \

  • ~/Documents/GitHub/parsec/conformance/baseline.json
    +git -C ~/Documents/GitHub/parsec diff conformance/baseline.json
    +rm -rf "$HOME/parsec-conformance"
    +```

+Read the diff: every count that went down is a case the release gives up, and the pull request says why. New suites appear in it as new entries.
+

3. Try the release without publishing

A manual run does everything but the tag check and the publishing:
@@ -262,6 +314,8 @@ Also look at docs.rs/parsec-html, which builds a

If something fails

+- A gate fails. Nothing was built or published. The job's summary says which suite, page or fuzz target. Fix it on main, delete the tag (as for the tag check, below) and tag the fixed commit. If a drop in a suite is meant, the baseline moves in a release pull request first.
+

  • The tag check fails. Nothing was published. Delete the tag (git -C ~/Documents/GitHub/parsec push origin :refs/tags/v0.1.0 and git -C ~/Documents/GitHub/parsec tag -d v0.1.0), fix what it says, and tag again.
  • A publish job fails after others succeeded. Fix the cause (usually a trusted publisher field that doesn't match), then gh run rerun RUN_ID --failed --repo parsec-html/parsec. Versions already up are skipped.
  • The crates.io checksum check fails. crates.io has a package that isn't the release's: someone published by hand, or the package isn't reproducible. Don't yank in a hurry; compare the two with scripts/package-twice.sh at the tag first.

</details>

**Tests.** `scripts/conformance-gate.test.sh` covers seven cases. Same counts, better counts, and more skips at the same passes all pass. One fewer passed, one more failed, and a missing suite each fail. A new suite passes and shows in the table. Each failing case would pass if the gate only checked `cargo test`'s status. The workflows pass `actionlint`, and `scripts/conformance-gate.sh` passes `shellcheck`. `boss --gate` passed locally on all 1,000 pages. Passed locally: `cargo test --workspace` (counts identical to CI's), `cargo fmt`, clippy for the bench with `-D warnings`, `cargo deny check licenses bans sources`.

A release went out whenever main was green, and green only means the
tests passed: a case moved under a skip, or a fuzz finding nobody had
looked at, still let the tag publish.

release.yml now runs three gates beside the tag check, and its build
waits on a `gates` job that passes only when all of them did:

- conformance: `cargo test --workspace`, then each suite's counts
  against conformance/baseline.json (the last release's counts), by
  scripts/conformance-gate.sh. Fewer passed or more failed in any
  suite, or a suite that wrote no counts, fails it.
- boss: `-- boss --gate`, all 1,000 pages through Parsec and
  html5ever, failing on a crash, a hang, a page without a document,
  or any page where the two count different elements.
- fuzz: five minutes on every target in crates/*/fuzz, listed from
  the manifests, one job each.

CI runs the same comparison on every pull request as a job that isn't
required, and its counts move from conformance/ to target/conformance,
since conformance/ now holds the baseline. bench/Cargo.lock still named
the crates at 0.1.0, which `--locked` refuses; it's at 0.2.0 now.
@jamesconwaydev jamesconwaydev added release About a release testing Trying it out and reporting back labels Nov 21, 2026

@foothillsdigital foothillsdigital left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shape is right. gates needs all three gates, and build checks needs.gates.result == 'success', so a skipped or failed gate can't be read as passing. The proof runs do what the description says. In run 44959465290 the gates took about 7 minutes before build, well inside the 30-minute wait in release-python.yml, release-npm.yml and release-capi.yml. conformance-gate.test.sh passes all seven cases.

One thing has to change: the gate fails open on a counts file it can't read. A counts file missing passed gives:

scripts/conformance-gate.sh: line 42: [: null: integer expression expected
scripts/conformance-gate.sh: line 45: [: null: integer expression expected
| a | null | 0 | 0 | same |
exit=0

The same happens when a file isn't valid JSON. jq runs inside < <(…), where set -e doesn't reach, so read gets empty values, and a [ that errors inside if counts as false. So a runner that writes a broken file, which is exactly the kind of silent regression this is meant to catch, passes the release. A check before the comparison closes it:

if ! jq -e '[.passed, .failed, .skipped] | all(type == "number")' "$file" > /dev/null 2>&1; then
  echo "| $suite | - | - | - | **unreadable counts** |"
  status=1
  continue
fi

Plus an eighth case in the test script with a file that has no passed.

Not blocking: the fuzz job runs cargo install cargo-fuzz --locked without --version, so a release builds with whatever cargo-fuzz is newest that day. fuzz.yml already does the same, and this job has only contents: read and no secrets, so it's low risk. Still, pinning both, like the actions, keeps a release reproducible.

jq ran inside < <(...), out of set -e's reach, and [ on an empty or
null value is false rather than an error, so a counts file missing
"passed", or not JSON at all, came out "same" and the release went
ahead. Every count is now checked to be a number first, in the baseline
and in each run's file, and an unreadable file fails the gate. Two more
cases in the gate's tests cover it.

cargo-fuzz is pinned to 0.13.2 in both the fuzz workflow and the release
gate, so a release doesn't build with whatever is newest that day.
@jamesconwaydev

Copy link
Copy Markdown
Collaborator Author

Fixed, thanks. That's exactly the silent failure this exists to catch, and I'd missed that set -e doesn't reach into < <(…).

  • Fails closed. Every count has to be a number before it's compared, in each run's file (your check, almost word for word) and in the baseline itself. An unreadable file prints **unreadable counts** and fails the gate. A broken file for a suite that isn't in the baseline yet fails too, since we can't tell what it is.
  • Tests. Two new cases: a file with no passed, and a file that isn't JSON. Nine cases now, all passing, and shellcheck is clean.
  • cargo-fuzz is pinned to 0.13.2 in both fuzz.yml and the release gate.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release About a release testing Trying it out and reporting back

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release gates: a release can't ship if a suite regressed

2 participants