[jwrosewell]

Problem

On Ubuntu the Selenium contract step reports a pass without running a test, so the browser contract has had no Linux coverage. And a pull request into rc runs all eight configurations, two of them on macOS, of which only five run at once across the whole organisation.

The false pass comes from ci/setup-environment.ps1, which deletes /usr/share/dotnet to free disk space. That leaves /usr/bin/dotnet as a link to nothing. PowerShell then opens the file with xdg-open rather than running it, which sets no exit code, so the $LASTEXITCODE check after dotnet test sees the 0 that curl left.

Cost, measured from run history

• In Ubuntu Node 24, Ubuntu Node 22 and the Ubuntu Node 22 nightly, the log shows curl refused, then three seconds later "Selenium contract tests passed" beside "xdg-open: file '/usr/bin/dotnet' does not exist", with no test count. On Windows Node 24 the same step ran 9 tests against each example.
• The cleanup in that Ubuntu job took 99 seconds and left 105GB free, so keeping .NET costs nothing that matters.
• A build where every job passed first time waited a median of 24.4 minutes and used 127.6 runner minutes. Five by-hand runs found nothing eligible and built rc in full, using 510 runner minutes, and configurations cancelled when another failed threw away 362.
• On 4 November the nightly (run 41809851364) queued macOS jobs for #23, a change to documentation alone, ahead of its own publish.

Change

1. ci/setup-environment.ps1 keeps .NET when it frees disk space.
2. ci/run-integration-tests.ps1 finds dotnet as a program and fails when it is missing, and fails when dotnet test reports no test that passed, as a run that finds no test also exits with 0.
3. A new ci/select-options.ps1 chooses the configurations for common-ci's selection step, which a companion common-ci change adds.
   • Standard set, for every change. Ubuntu_Node_24 and Ubuntu_Node_22, as each builds its own native addon, with the Selenium contract on Ubuntu_Node_24 alone.
   • A native package or the cxx submodule, changed in more than its comments, adds Windows_Node_24 and Ubuntu_ARM_Node_24.
   • A change outside the package folders, such as ci or the root package.json, adds every configuration but macOS.
   • Only when rc itself or a release is built do the two macOS configurations and the performance tests run.
4. The Pull Requests workflow passes fail-fast: false and build-without-pull-requests: false, and gains a full-matrix box for a full build of a pull request by hand.
5. The nightly runs its publish first and builds pull requests after it, as the publish builds main, which the pull requests into rc do not change. It starts at 02:30 rather than 03:00, when the Java and Python nightlies also started and wanted macOS runners.
6. A new release-pull-request.yml builds the release pull request from rc into main on every configuration, as a dry run that merges nothing.
7. The repository notes describe the selection and the .NET dependency.

Expected saving

Change	Configurations	Wait	Runner minutes
Today, any change	8	24.4	127.6
Documentation alone	0	0	0
Code outside native packages	2	about 17	about 31
A native package or the cxx submodule	4	about 23, set by Windows	about 65
CI or other files outside the package folders	6	about 23	about 97

Ubuntu_Node_24 gains about 2 minutes for a contract that really runs, being the 49 and 39 seconds of tests in the Windows job. Applied to the changed files of the 17 pull requests opened since 31 October, the mean runner minutes fall from 128 to about 68, and no pull request build takes a macOS runner. With the fail-fast setting and the by-hand builds of rc it stops, that is about 370 runner minutes a day over those days. The release pull request adds one full build, about 128 runner minutes, for each release.

How it was checked

• The cause was read from the three job logs above and ci/setup-environment.ps1.
• Every changed PowerShell file parses, and every changed workflow parses as YAML. The jobs moved in nightly-pipeline.yml are identical to the old ones apart from their order, needs, if, the new input and the schedule.
• The selection rules were applied, in a copy, to the changed files of every pull request since 31 October, which gives the table above.
• e1a3546c merges rc at 24592ac. The one conflict was AGENTS.md, where rc had added the Releases, Versions and timing sections after the CI section, and both are kept.
• Build 43478047802 on the head e1a3546 passed. All seven Setting 'PullRequestSha' lines are e1a3546. ci/select-options.ps1 saw a change outside the package folders and chose every entry but macOS, so six jobs ran, each with 1,355 unit tests passed and 2 skipped and 132 integration tests passed and 4 skipped. The Selenium contract ran on Ubuntu_Node_24 alone, "Passed! - Failed: 0, Passed: 9" against the cloud example and again against the on-premise example, with no xdg-open line, and the other five said "Another entry runs the Selenium contract tests for this pull request."
• For comparison, build 43477197935 of another draft on today's rc still prints "Selenium contract tests passed" beside "xdg-open: file '/usr/bin/dotnet' does not exist" in both Ubuntu jobs on Node 22 and 24, with no test count.

ad948739, after that build, changes one sentence of AGENTS.md alone, so that it says a native change adds Node 24 on Windows and on Linux ARM.

This draft changes only CI scripts, workflows and their notes, so under James Rosewell's decision of 11:35 UTC on 12 November it needs only its quick checks. The build above was run as well because the Selenium contract can now fail on Linux.

What it gives up

On a pull request, macOS, Node 22 on Windows and ARM unless a change falls outside the package folders, Node 24 on Windows and ARM unless native code changes, the Selenium contract on all but one configuration, and performance. A fault that shows only there is found by the nightly build of rc or by the release pull request build, before anything reaches main. The contract now really runs on Linux, which it did not before.

Merge order

The common-ci change it needs is on common-ci main (1c5c0e0), which has full-matrix, fail-fast and build-without-pull-requests in nightly-pull-requests.yml, the ci/select-options.ps1 step in nightly-pull-request.configure-pr.ps1 lines 36 to 41 and release-pull-request.yml. So it can merge into rc and reach main with the next release.

What remains

• The "Cloud 51Did Tests" filter their pull_request trigger on main alone (cloud-51did.yml lines 32 to 36 at 24592acb), so no pull request into rc has run them since changes moved to rc. Adding rc is left for a change of its own, because changing that workflow runs it on the pull request.
• A Selenium contract that can now fail on Linux may show up faults that the false pass hid.