[jwrosewell]

Problem

The first Selenium contract test of a job starts Chrome, and Chrome's first start on a fresh runner can take longer than the test's 60 second limit. In full build 43270028176 the job Ubuntu_PHP_7.4 failed that way, "The HTTP request to the remote WebDriver server for URL http://localhost:42445/session timed out after 60 seconds", in the first test. 51degrees-java#124 found the same wait failing on Intel macOS runners and measured Chrome's first start there.

Firefox has the same cost. In full build 43520416596, which started Chrome once and not Firefox, the first run of SessionStorageCache_FireFox (True) in each job took 10 to 32 seconds, and the same test took 2 to 4 seconds in the job's second contract run.

A test that fails on timing is fixed rather than run again, either with a limit that fits the slowest runner or by finding why it is slow. Here the slow part is not the test. It is each browser starting for the first time on a new machine, so this draft moves those starts out of the tests.

What changed

• ci/start-browsers-once.ps1 starts the Chrome driver, Chrome, the Firefox driver and Firefox once, waits up to 300 seconds for each, and prints how long each took. A start that runs over is stopped and one that fails is reported, both as warnings, and the tests then run as they would have without it.
  • Chrome is started as in the script of 51degrees-java#124 at 6a757dad, found the same way (CHROME_BIN, then the path, then where Chrome installs itself) and with the same arguments.
  • Firefox and its driver are found the way the shared suite's Helpers/BrowserDrivers.cs finds them, being FIREFOX_BIN and GECKOWEBDRIVER first, then firefox and geckodriver on the path. The driver is run with --version. Firefox takes a headless screenshot of about:blank with a profile of its own, which is then deleted. On Windows it is given --wait-for-browser, because Firefox's launcher there can hand the start to a second process and exit at once, and that option makes it wait for the browser, so the time includes the start.
  • The finding, timing and reporting of java#124's script are kept and shared by both browsers, so the file is renamed from start-chrome-once.ps1.
• The root ci/run-integration-tests.ps1 runs it once before any package's integration tests, so the first starts fall outside the limit of every contract test, cloud and on-premise.
• The Device Detection cloud and on-premise scripts run the contract tests with --logger "console;verbosity=normal", so each test's time is printed, passed or failed.

The call sits in the root script, not in a package's ci/, because a package directory is published to its mirror as it is. The two package scripts changed sit under device-detection/cloud/ci/ and device-detection/onpremise/ci/, which the release plan of #37 ignores, so this draft releases no package.

How it was checked

This draft changes CI scripts only, with no package code. James Rosewell's decision of 12 November for such changes is "rebuild and include, but the change doesn't impact the code so no need to run the full builds", so it needs green quick checks on the head and a review. Full builds ran anyway, to measure each browser's first start on every runner.

Full build 43533876757 ran on the head f6140ff and passed, with 9 of 9 Setting 'PullRequestSha' lines on f6140ff and "Left out 5 macOS entries". Every contract run printed "Passed: 9", and no warning was printed.

Runner	Chrome driver	Chrome	Firefox driver	Firefox	First Firefox test, before	First Firefox test, now
Ubuntu, PHP 7.4	0.7 s	26.3 s	0.3 s	24.8 s	not printed	12 s
Ubuntu, PHP 8.0	0.8 s	12.1 s	0.1 s	8.7 s	10 s	7 s
Ubuntu, PHP 8.1	0.5 s	19.9 s	0.2 s	18 s	13 s	9 s
Ubuntu, PHP 8.2	0.4 s	14.7 s	0.2 s	11.7 s	32 s	4 s
Ubuntu, PHP 8.3	0.9 s	7.1 s	0.6 s	24.9 s	17 s	9 s
Ubuntu, PHP 8.4	0.4 s	9.1 s	0.3 s	16 s	21 s	12 s
Ubuntu, PHP 8.5	0.7 s	1.3 s	0.4 s	1.6 s	29 s	5 s
Windows, PHP 8.3	0.4 s	6.8 s	0.2 s	2.9 s	16 s	14 s

The first four columns are the first starts this script made, outside every test, and they took up to 26.3 seconds. "First Firefox test" is SessionStorageCache_FireFox (True) in the job's first contract run, before in build 43520416596 and now in build 43533876757. It is the slowest contract test in every job of the new build, and the slowest is 14 seconds, against its 60 second limit. Its second run in the same job takes 2 to 4 seconds, so a few seconds of each new Firefox session's own start are left in the first test, which a test cannot avoid because each session makes a new profile.

The Chrome start of java#124's script has its own red and green runs on 51degrees-java#124, red 43518883531 (a Chrome that exits with code 3 was reported as started) and green 43518903716. Build 43520416596 above is this draft's earlier head 733201e, which warmed only Chrome, and passed with 9 of 9 Setting 'PullRequestSha' lines on 733201e.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

The head

f6140ff is b794a13 plus the Firefox commit. b794a13 is 733201e with rc at 6d95eee merged in, with a merge commit and no conflict. The quick checks on f6140ff are green, Environment variable names 43533875091, Only rc into main 43533875510 and UTM link lint 43533875207.

What remains

1. The macOS start times come from the release build of the rc head, which runs with macOS.
2. common-ci#324 adds CHROME_BIN, CHROMEWEBDRIVER, FIREFOX_BIN and GECKOWEBDRIVER, which the script reads, to the one list of environment variable names, with the other browser names the shared Selenium suite reads.
3. 51degrees-java#124 starts Chrome only, and Java runs the same shared suite, so it can take this script once this is reviewed.