[jwrosewell]

Problem

A browser's first start in a run of this suite can take far longer than the starts after it, and while it does, the first test that uses the browser spends its own time on it. The Selenium .NET client gives each command 60 seconds, so a slow first start can fail a test that has nothing wrong with it.

• 51degrees-node#98, build 43560055156 attempt 1, Windows Node 24. Example_RendersRealDetectionResult failed with "session not created from disconnected: unable to connect to renderer" from new ChromeDriver in Helpers/BrowserDrivers.cs line 130.
• 51degrees-python#134, build 43590775623, Ubuntu Python 3.13. Example_RendersClientSideOverridesOnThePage failed with "Driver service process exited unexpectedly before initialization completed".
• 51degrees-rust#416 read every case's time in the 30 most recent runs of the Rust examples on ubuntu-latest. Only the first case each browser ran was slow. In the cloud example the first Chrome case took a median of 11 s and up to 174 s, and the first Firefox case a median of 21 s and up to 62 s, where the cases after them took a few seconds. The 174 s case failed when its page load passed the 60 second limit.

Each language worked round this in its own CI, by starting the browser once before the suite runs (51degrees-java#124, 51degrees-rust#416, 51degrees-dotnet#207 for .NET's own browser tests). Doing it here gives every language's run of this suite the same, through the driver the suite itself uses.

What changed

• BrowserDrivers.StartOnce runs a browser's first start the first time the browser is asked for in a run, and never again. The start creates the driver with a limit of 300 seconds for each command (FirstStartTimeoutSeconds), loads about:blank and quits. It writes "Started Chrome once before its first test in N s." to the output of the test that asked. Whatever stops the start is written with the first line of its reason and is not thrown, so the test then starts the browser as it would have and meets any real fault itself.
• CreateChrome and CreateFirefox call it before they create the test's driver, with the same driver, binary and options, whether the machine provides the driver or Selenium Manager fetches it. A remote Selenium (SELENIUM_URL) is not started first, because a grid keeps its browsers running.
• Edge is not started first, because headless Edge on the Ubuntu runners hangs on its first page (the reason Edge_RunsThePage is ignored), which would hold the run for the full 300 seconds. No Contract test uses Edge.
• FirstStarts gives the line written for each browser, for the tests.
• README.md says so under "How a run is wired".

Tests

New Selenium/FirstStartTests.cs. They reach StartOnce, FirstStarts and FirstStartTimeoutSeconds by name, so that on rc they fail when they run rather than stopping the build.

• StartOnce_StartsEachBrowserOnce and StartOnce_AStartThatFails_IsReportedAndNotThrown use a stand-in start and need no browser, so they run in this repository's first CI job. A browser asked for twice is started once and given the 300 second limit, and a start that throws is written, not thrown, and not tried again.
• Chrome_StartsOnceBeforeItsFirstTest and Firefox_StartsOnceBeforeItsFirstTest, in the Browser category, create the browser twice, load a page each time and check the first start was made. They write how long each of the two took.

Both runs used a temporary branch holding the commits and one workflow that builds the suite and runs these tests and DriverSmokeTests on ubuntu-latest, ubuntu-22.04, ubuntu-22.04-arm and windows-latest. The branches are deleted.

• Red, run 43614291946, the tests on rc. On each of the four runners the four new tests failed ("BrowserDrivers has no StartOnce" and "has no FirstStarts") and the two smoke tests passed.
• Green, run 43614318548, with the change. Six of seven passed on each runner, and Edge_RunsThePage was skipped as it always is.

The times the green run wrote, in seconds:

Runner	Chrome, first start	Chrome, then	Firefox, first start	Firefox, then
ubuntu-latest	1.4	0.4 and 0.4	2.9	2.2 and 2.1
ubuntu-22.04	6.6	0.4 and 0.4	6.1	2.3 and 2.2
ubuntu-22.04-arm	7.4	0.4 and 0.4	20.2	2.2 and 2.3
windows-latest	9.9	2.1 and 1.8	13.6	8.3 and 8.4

"Then" is a later start of the same browser with a page loaded, as a test makes it. So in this run every first start took longer than the starts after it, by up to 18 seconds, and none came near 60 seconds. The longer first starts recorded in the other repositories did not happen here, and this run shows only that the first start is now made before the first test and timed.

Build

This repository's own checks on this pull request build the suite, run the tests that need no browser, and start Chrome and Firefox on its three Ubuntu runners.

What remains

• The language repositories run this suite from main, so the change reaches them when rc is released into main. main holds the two release commits of 11 November that rc does not, so main is merged back into rc first.
• macOS was not measured here, because this repository's CI has no macOS runner.