[jwrosewell]

Problem

ci/packages.ps1 works out the products a pull request touches from its diff with main. A pull request into rc is checked out with rc merged in, so that diff also holds every change on rc not yet released. On 4 November rc was 29 files ahead of main, including pipeline/, and a change to pipeline/ touches every product, so every pull request selected every product until the next release.

And a pull request into rc runs all 15 configurations in ci/options.json, five of them on macOS, of which only five run at once across the whole organisation.

Cost, measured from run history

• No pull request build ran between 3 and 4 November, so the nearest measure is the publish test jobs, which took a median of 20 to 22 minutes on Ubuntu, 29 to 35 on Windows and 25 to 30 on macOS for each Python version.
• On 4 November the nightly (run 41807348984) built the automated submodule update #26 on all 15 configurations, holding a macOS runner while other repositories' publishes waited for one.
• The Java and Node.js nightlies also start at 03:00 (nightly-pipeline.yml on each repository's main, read at 18:20 UTC on 13 November), and all three want macOS runners.

Change

1. Get-AffectedProducts in ci/packages.ps1 takes the branch the pull request merges into, which common-ci passes as Branch. ci/dispatch.ps1 passes it on when it has no list of products, and every root step declares it. A publish or a build of the branch itself still selects every product.
2. A new ci/select-options.ps1 chooses the configurations for common-ci's steps/select-options.ps1, which common-ci#309 added and which is on common-ci's main since common-ci's release of 12 November. Within each configuration ci/packages.ps1 still runs only the products the change touches.
   • Standard set, for every change. Ubuntu_Python_3.14 and Ubuntu_Python_3.10, the newest and the oldest, with the Selenium contract on 3.14 alone.
   • A change to an on-premise package or the cxx submodule, in more than its comments, adds Windows_Python_3.14, where the native module is built differently. The data files come in through cxx, so a change to them is a change to cxx.
   • A change to a setup.py, setup.cfg or pyproject.toml adds Python 3.11 to 3.13 on Linux.
   • A change outside the product folders or to a ci folder adds every configuration but macOS.
   • Only when rc itself or a release is built do the five macOS configurations and the performance tests run.
3. The root ci/run-integration-tests.ps1 passes the Selenium choice to device-detection/ci/run-integration-tests.ps1, which skips the contract where SeleniumContract is false. It is true by default.
4. The Pull Requests workflow passes fail-fast: false and build-without-pull-requests: false, and gains a full-matrix box for a full build of a pull request by hand.
5. The nightly runs its publish alongside the submodule update and builds pull requests after both, as the publish builds main, which the pull requests into rc do not change. It starts at 05:00 rather than 03:00, after the Java and Node.js nightlies, and the schedule's comment says why.
6. A new release-pull-request.yml builds the release pull request from rc into main on every configuration, macOS included, through common-ci's release-pull-request.yml, as a dry run that merges nothing.
7. The repository notes describe the selection.
8. ci/select-options.Tests.ps1 is new. It reads the real ci/options.json as common-ci does, as hashtables, and checks the choice for a package's code, an on-premise package, cxx, a change only to an on-premise file's comments, a setup.py, a ci folder, the root ci folder and a workflow file, that the Selenium contract runs on Python 3.14 alone with no performance tests, and that no pull request builds macOS.

Expected saving

Change	Configurations	Wait	Runner minutes
Today, any change	15	not measured	about 410, from the publish test medians
Documentation alone	0	0	0
Code in a product folder	2	about 22	about 43
An on-premise package or the cxx submodule	3	about 29, set by Windows	about 72
Packaging	5	about 22	about 106
CI or files outside the product folders	10	about 35	about 271

These use the publish test medians, which install the built packages and run the integration tests, so they are not the same work as a pull request build. Within each configuration the base fix also stops every product being built for a change to one.

How it was checked

• rc at df0949d7 is merged in at 4cc8f18. The one conflict was in ci/dispatch.ps1, where rc had given Invoke-ProductStep a list of products to run, which the prefetch of assets uses, and this draft the branch to measure from. Both are kept, and the branch is used only when no list is given.
• The new tests found a defect in this draft's first ci/select-options.ps1. PowerShell names are not case sensitive, so the flag $native and the list of entries $Native were one variable, the list became True, and True -contains <name> holds for every name, so any change to native code built all ten Ubuntu and Windows entries. Run 43480673571, at 6ce34bd, failed adds Windows for a change to an on-premise package and adds Windows for a change to the cxx submodule on ubuntu-latest and windows-latest, Tests Passed: 138, Failed: 2. Commit 2e8d370b renames the flags to $nativeChanged, $setupChanged and $otherChanged.
• Green, this head, run 43486193686, passed on both, Tests Passed: 140, Failed: 0. Both runs are temporary branches holding a workflow that checks out common-ci's main beside this repository and runs every ci/*.Tests.ps1 with Pester.
• Quick checks passed on this head, being run 43486190045 (environment variable names), run 43486190412 (only rc into main) and run 43486190046 (link campaign tags).
• This draft changes only CI scripts, workflows, their tests and AGENTS.md, with no package code. James Rosewell decided at about 11:35 UTC on 12 November, for such a change, "rebuild and include, but the change doesn't impact the code so no need to run the full builds."

What it gives up

On a pull request, macOS, Windows unless native code changes, Python 3.11 to 3.13 unless the packaging changes, the Selenium contract on all but one configuration, and performance. A fault that shows only there is found by the nightly build of rc or by the release pull request build, before anything reaches main.

Merge order and the full build tool

The coordinator's full build tool copies the with: lines of this repository's pull-requests.yml into a temporary workflow that calls common-ci's nightly-pull-request.yml. Once this draft is on rc, it would copy build-without-pull-requests: false, which nightly-pull-request.yml does not take, and full-matrix: ${{ inputs.full-matrix || false }}, which is false on a push, so every draft's full build would either fail to start or be cut down to the configurations the change needs. 51degrees-cxx#99 has the same lines. So this draft merges after the tool leaves build-without-pull-requests out and passes full-matrix: true, and after the full builds of the drafts that merge before it.

What remains

• A first measured pull request build, which comes with the first pull request into rc after this merges.

The last release into main was #114, merged at 06:22 UTC on 13 November.