Repository navigation
ci(release): a release could ship with fewer conformance cases passing - #212
jamesconwaydev wants to merge 3 commits into
Conversation
A release went out whenever main was green, and green only means the tests passed: a case moved under a skip, or a fuzz finding nobody had looked at, still let the tag publish. release.yml now runs three gates beside the tag check, and its build waits on a `gates` job that passes only when all of them did: - conformance: `cargo test --workspace`, then each suite's counts against conformance/baseline.json (the last release's counts), by scripts/conformance-gate.sh. Fewer passed or more failed in any suite, or a suite that wrote no counts, fails it. - boss: `-- boss --gate`, all 1,000 pages through Parsec and html5ever, failing on a crash, a hang, a page without a document, or any page where the two count different elements. - fuzz: five minutes on every target in crates/*/fuzz, listed from the manifests, one job each. CI runs the same comparison on every pull request as a job that isn't required, and its counts move from conformance/ to target/conformance, since conformance/ now holds the baseline. bench/Cargo.lock still named the crates at 0.1.0, which `--locked` refuses; it's at 0.2.0 now.
foothillsdigital
left a comment
There was a problem hiding this comment.
The shape is right. gates needs all three gates, and build checks needs.gates.result == 'success', so a skipped or failed gate can't be read as passing. The proof runs do what the description says. In run 44959465290 the gates took about 7 minutes before build, well inside the 30-minute wait in release-python.yml, release-npm.yml and release-capi.yml. conformance-gate.test.sh passes all seven cases.
One thing has to change: the gate fails open on a counts file it can't read. A counts file missing passed gives:
scripts/conformance-gate.sh: line 42: [: null: integer expression expected
scripts/conformance-gate.sh: line 45: [: null: integer expression expected
| a | null | 0 | 0 | same |
exit=0
The same happens when a file isn't valid JSON. jq runs inside < <(…), where set -e doesn't reach, so read gets empty values, and a [ that errors inside if counts as false. So a runner that writes a broken file, which is exactly the kind of silent regression this is meant to catch, passes the release. A check before the comparison closes it:
if ! jq -e '[.passed, .failed, .skipped] | all(type == "number")' "$file" > /dev/null 2>&1; then
echo "| $suite | - | - | - | **unreadable counts** |"
status=1
continue
fiPlus an eighth case in the test script with a file that has no passed.
Not blocking: the fuzz job runs cargo install cargo-fuzz --locked without --version, so a release builds with whatever cargo-fuzz is newest that day. fuzz.yml already does the same, and this job has only contents: read and no secrets, so it's low risk. Still, pinning both, like the actions, keeps a release reproducible.
jq ran inside < <(...), out of set -e's reach, and [ on an empty or null value is false rather than an error, so a counts file missing "passed", or not JSON at all, came out "same" and the release went ahead. Every count is now checked to be a number first, in the baseline and in each run's file, and an unreadable file fails the gate. Two more cases in the gate's tests cover it. cargo-fuzz is pinned to 0.13.2 in both the fuzz workflow and the release gate, so a release doesn't build with whatever is newest that day.
|
Fixed, thanks. That's exactly the silent failure this exists to catch, and I'd missed that
|
# Conflicts: # CONTRIBUTING.md
Fixes #202.
Problem. A release was tagged whenever
mainwas green, and green only means the tests passed. A case that moves under a skip, or out of the loop, drops a conformance count and still passes. The final boss ran only as a 300-page smoke test, and fuzzing ran nightly with nobody required to look at it before a tag.Change.
release.ymlruns three gates beside the tag check.build(and sopublish,crates-io, and the Python, npm and C workflows, which wait for that GitHub release) needs agatesjob, which passes only when all three did:conformance:cargo test --workspace --no-fail-fastwithCONFORMANCE_DIR, thenscripts/conformance-gate.sh conformance/baseline.json. The release fails if any suite passes fewer cases or fails more, or if a suite in the baseline wrote no counts. The table goes into the run's summary.boss:cargo run --release --locked --manifest-path bench/Cargo.toml -- boss --gate, a new mode that runs all 1,000 pages (giants up to 30 MB included) once through Parsec and html5ever. It fails on a crash, a page over 10 s, a refused page, a page without a document, or any page where the two element counts differ. It takes about a minute on the runner. Locally all 1,000 pages give 0 differences.fuzz: every cargo-fuzz target, listed fromcrates/*/fuzz/Cargo.tomlbycargo metadata, gets five minutes on nightly in its own matrix job. A finding fails the release and is uploaded. A new target, such asparsec-js's, is picked up without a change to the workflow.conformance/baseline.json: each suite'spassed,failedandskipped, from theconformanceartifact ofmain's CI run at 7006d31 (32 suites; macOS gives the same counts). test262 and xmlconf aren't onmainyet, so they're absent. The gate lists a suite with no entry as "new" and lets it pass, and the release pull request adds it.ci.yml: a newconformancejob runs the same comparison on every pull request from the test job's artifact. It isn't a required check, but a drop shows red on the pull request that made it. The test job now writes its counts totarget/conformance, sinceconformance/holds the baseline. The lint job runsscripts/conformance-gate.test.sh.bench/Cargo.lockstill named the crates at 0.1.0 after the 0.2.0 bump, which--lockedrefuses. It's at 0.2.0 now.Proof. release run 44960154608 is a manual run of
release.ymlon the throwaway branchrelease-gates-proof, which skips one passingwpt-url-parsingcase.cargo teststays green, and the gate turns the release red:gates,build,publishandcrates-iowere skipped. On this branch, release run 44959465290 passes every gate: conformance in 4 min, boss in 1 min, both fuzz targets in 6.5 min, all at once. The same drop run locally:Wiki. I haven't pushed the wiki. This patch to
Releasing.mdadds a "The gates" section, with how the baseline moves, how test262 and xmlconf join it, and speed as a manual gate (bench at load under 3 against the previous tag, with any slowdown over 5% explained in the release notes). It also adds the baseline update to step 2 and "A gate fails" under "If something fails":Releasing.md patch
-### 2. Set the version and the changelog
+### 2. Set the version, the changelog and the baseline
For 0.1.0 this is done: every version is already 0.1.0. For the next one, change
versionin~/Documents/GitHub/parsec/Cargo.toml([workspace.package]), theversion = "…"requirements incrates/*/Cargo.toml,bindings/python/Cargo.toml,bindings/wasm/Cargo.toml,bindings/wasm/npm/package.jsonand thePARSEC_VERSIONlines incrates/parsec-capi/cbindgen.toml(then run the cbindgen command at its top, soinclude/parsec.hfollows; a test fails if they disagree), and add the release toCHANGELOG.md, in a pull request like any other.+The same pull request moves
conformance/baseline.jsontomain's counts, from theconformanceartifact ofmain's latest CI run:+
+```sh
+run=$(gh run list --repo parsec-html/parsec --branch main --workflow ci.yml --status success --limit 1 --json databaseId -q '.[0].databaseId')
+gh run download "$run" --repo parsec-html/parsec -n conformance -D "$HOME/parsec-conformance"
+jq -s 'map({key: .suite, value: {passed, failed, skipped}}) | sort_by(.key) | from_entries' "$HOME"/parsec-conformance/*.json \
+Read the diff: every count that went down is a case the release gives up, and the pull request says why. New suites appear in it as new entries.
+
3. Try the release without publishing
A manual run does everything but the tag check and the publishing:
@@ -262,6 +314,8 @@ Also look at docs.rs/parsec-html, which builds a
If something fails
+- A gate fails. Nothing was built or published. The job's summary says which suite, page or fuzz target. Fix it on
main, delete the tag (as for the tag check, below) and tag the fixed commit. If a drop in a suite is meant, the baseline moves in a release pull request first.+
git -C ~/Documents/GitHub/parsec push origin :refs/tags/v0.1.0andgit -C ~/Documents/GitHub/parsec tag -d v0.1.0), fix what it says, and tag again.gh run rerun RUN_ID --failed --repo parsec-html/parsec. Versions already up are skipped.scripts/package-twice.shat the tag first.