[jwrosewell]

Closes #126.

Problem

The Selenium contract tests timed out on the Intel macOS runners when the first test of the run, ExampleClientSideOverridePageContentTests.Example_RendersClientSideOverridesOnThePage, waited longer than Selenium's 60 second command limit in GoToUrl. #124 starts Chrome once, directly, before the tests, which takes the first start of Chrome (156.1 to 162.2 seconds on those runners in its build) outside the tests. Even so, in full build 43442211049 the first test still took 17, 23 and 34 seconds on the three Intel macOS runners, against 3 to 5 seconds for the same test's other runs, as #126 sets out. Part of the first cost came with the first session through the Chrome driver, which a direct start of Chrome does not make.

What changed and why

ci/start-chrome-once.ps1 (a232305f), which the contract tests' step in ci/run-integration-tests.ps1 runs first, now also runs one session through the Chrome driver after starting Chrome directly, as the first contract test does.

• It starts the driver the tests find (CHROMEWEBDRIVER, then the path) on a free port on the loopback address and waits until its /status says it is ready.
• It asks for a session as the first test does, with --headless and acceptInsecureCerts, and names the browser only when the tests would (CHROME_BIN, then a browser on the path), as BrowserDrivers.CreateChrome at e8605df3 and the test's options do.
• It loads about:blank, ends the session and stops the driver, and prints how long each part took.

Each command has the script's existing limit of 300 seconds. A part that fails is a warning and the tests then run as they would have without it, and the driver and the browser profile are removed whatever happens, as for the direct start. It is a CI script only, so it changes no package and releases nothing.

How it was measured

A throwaway run, 43588616992 on tmp/java-chrome-session, ran the script twice on each of the four images CI uses for Chrome. The first session through the driver took 19.1 seconds to start on macos-15-intel, after Chrome's direct first start of 171.8 seconds, and 1.9 seconds the second time. It took 3.9 seconds on macos-15, 2 seconds on Windows and 0.2 seconds on Ubuntu, with no warning on any.

Full build

Run 43590236784 built a232305 merged with rc (1c99b72) on all fifteen configurations, macOS included, as #126 asks for a build with macOS, and every job passed, Compare Performance included. Each of its sixteen Setting 'PullRequestSha' lines names a232305, and its Configure job left nothing out.

The first contract test now takes what its other runs take. Each row is that test's four runs on one runner, in the order they finished, the first being the first test of the run.

Runner	Chrome's direct first start	First driver session started in	The test's four runs
MacOS_X64_Java_11	173.4 s	26.3 s	3, 3, 3 and 3 s
MacOS_X64_Java_17	149.6 s	24.1 s	6, 3, 3 and 3 s
MacOS_X64_Java_21	149.8 s	14.3 s	4, 3, 3 and 3 s
MacOS_Java_11	6.4 s	2.4 s	5, 3, 2 and 1 s
MacOS_Java_17	stopped at 300 s	3.3 s	3, 1, 3 and 3 s
MacOS_Java_21	7.1 s	3.8 s	5, 2, 2 and 2 s
Windows, Java 8 to 21	3.1 to 9.6 s	1.4 to 1.9 s	2 to 3 s
Ubuntu, Java 8 to 21	1.1 to 17.7 s	0.3 to 0.6 s	2 s, then 1 s

On the Intel macOS runners the first test took 3, 6 and 4 seconds against 3 seconds for the runs after it, where #124's build measured 17, 23 and 34 seconds, so the first session through the driver, which now takes 14.3 to 26.3 seconds before the tests, was the cost left.

What remains

• On MacOS_Java_17 the direct start of Chrome, which Start Chrome once before the Selenium contract tests #124 added, did not finish within 300 seconds and was stopped with its warning. The driver session after it started in 3.3 seconds and the tests passed, but the job spent five minutes waiting. It happened once in this build's six macOS jobs, and Start Chrome once before the Selenium contract tests #124's build saw no stop. If it recurs, the direct start could be dropped in favour of the driver session alone, which then makes Chrome's first start itself.