[jwrosewell]

Closes #93. Also makes every on-premise Device Detection example, and the IP Intelligence web examples, run from a fresh clone with nothing set.

The problem

Loading FiftyOneDegreesHashEngine.so stops PHP before any script runs when FiftyOneDegreesHashEngine.data_file names no data file that exists. With the setting left out, PHP stops with "The data file '/usr/lib/php5/51Degrees.hash' could not be found", an uncaught StatusCodeException and exit code 134. This was found in #88's run 43499740734, on PHP 8.3 and 8.5. The %minit block of hash-php.i builds the engine when PHP starts with nothing catching the error, so a missing file, an empty setting, a file that is not a data file, or settings the engine cannot use all stop PHP the same way. The first red run below shows each of them ending with exit code 134.

So a fresh clone cannot run any on-premise Device Detection example without first downloading the Lite file by hand and naming it in php.ini. downloadMissingDataFile was only reached from useDataFileFromEnv, and only when _51DEGREES_DD_PATH is set, and no console example called either. IP Intelligence already runs its console examples again in a PHP started with the file it finds or downloads (ip-intelligence/examples/classes/ExampleUtils.php). Neither package's web examples can do that, because a request cannot start PHP again, so the Device Detection readme asked for php.ini first, and the IP Intelligence readme asks for the data file in php.ini for its web pages too.

What the specification says

This is required-examples.md, lines 91 to 97, under "On-Premise Device Detection".

Examples MUST run out of the box with the free data files. Given no arguments or settings, an example uses the Lite Data File, 51Degrees-LiteV4.1.hash, which the data repository links in its device-detection folder. An example looks for the file and, when it is missing, downloads it with the data repository's get-data-files script, so a fresh clone of a language repository runs every example with nothing set. A paid data file can be given as an argument or setting instead.

"Every example" includes the web examples, and a start command that the person has to complete with a path is a setting. So a web example also has to start from a fresh clone with nothing set, which is what the launchers below do.

What changed and why

1. The extension always lets PHP start (hash-php.i). It builds the engine only when the setting names a file that exists, as ipi-php.i does, and catches a failure to build it. With no engine, FiftyOneDegreesHashEngine::engine_get() gives null. A setting naming a missing file, or a file or settings the engine cannot be built from, gives a PHP warning when PHP starts with the path and the reason. Leaving the setting out, with no file at the default path, gives no warning.
2. The setting's name and its default value are unchanged. A patch release never changes what an existing installation loads, so an installation that relies on the default /usr/lib/php5/51Degrees.hash loads the same file as before.
3. DeviceDetectionOnPremise refuses with a message (DeviceDetectionOnPremise.php, Messages.php), as IpIntelligenceOnPremise does, rather than failing later with a null. With no engine, the message says the setting is not set, or names the missing file, or names the file the engine could not be built from and points at the startup warning, and then says to set FiftyOneDegreesHashEngine.data_file in php.ini. Without the extension it says the extension is not loaded, where it used to end in "Class not found". The new public names are Messages::EXTENSION_NOT_LOADED, NO_ENGINE, DATA_FILE_NOT_SET, DATA_FILE_MISSING, DATA_FILE_NOT_LOADED and Constants::DATA_FILE_SETTING, and none is removed or renamed.
4. The committed wrapper is regenerated. hash_wrap.cxx was generated with SWIG 4.5.1 by generate-wrapper.sh on a runner, building SWIG as php-wrapper-check.yml does (run 43535118510, job "Generate the Device Detection wrapper", whose --check then passed). It differs from the one on rc only in the two blocks hash-php.i changes. php_FiftyOneDegreesHashEngine.h came out the same.
5. Every on-premise Device Detection example calls ExampleUtils::startWithEngine first (ExampleUtils.php), as the IP Intelligence console examples do. When the extension built an engine, the example carries on. Otherwise a console example takes the data file its first argument names, then the one _51DEGREES_DD_PATH names, then the Lite file in cxx/data/device-detection, downloaded with the data repository's script when it is missing, and runs again in a PHP started with -d FiftyOneDegreesHashEngine.data_file=<file>. It says so on standard error, with the command, and says to set the data file in php.ini so that PHP builds the engine once. A named file that does not exist is refused with a message rather than replaced by the Lite file. The console examples are gettingStartedConsole.php, metadataConsole.php, matchMetrics.php, failureToMatch.php and manualDataUpdate.php. A web example cannot start PHP again, so gettingStartedWeb.php and userAgentClientHints-Web.php, started some other way with no engine, answer with HTTP 500 and a page giving the launcher command, the php.ini setting, and where the free file is or the command that downloads it.
6. Launchers start the web examples with nothing set. device-detection/onpremise/examples/onpremise/serve.php and ip-intelligence/examples/onpremise/serve.php take the example's name, then an optional data file and port (3000 unless given). Each finds the data file as its package's console examples do, the free Lite file downloaded when there is none, or uses php.ini's file when PHP built an engine from it and no other file is named. It then runs php -d <setting>=<file> -S localhost:<port> in the foreground, printing the address and saying to set the file in php.ini for real use. The server keeps the options the launcher was started with, such as -d extension=..., where the system gives PHP's own command line, as Linux does. The Device Detection launcher serves gettingStartedWeb.php or userAgentClientHints-Web.php as the router script. The IP Intelligence launcher serves the web or web-mixed folder, and for web-mixed also gives the Device Detection data file that _51DEGREES_DD_PATH names, or else the Lite file, when PHP built no Device Detection engine. For IP Intelligence this draft adds only the launcher, its tests and the start commands, and leaves the web pages as #88 and #98 made them.
7. Docs. device-detection/onpremise/readme.md says what happens with no usable data file, that the graph settings no longer stop PHP, how the examples find the file, and starts the web example with the launcher. ip-intelligence/readme.md gives the launcher command for its web examples. AGENTS.md runs the Device Detection console example from a fresh clone with only -d extension=..., gives the launcher commands for both packages' web examples, including the mixed page in place of Load both on-premise extensions in one PHP, and run the mixed examples on the free data files #88's -S command that named both data files, keeps the php.ini way for building the engine once, updates two pitfalls, and says in "What CI tests" which tests run the examples with nothing set.
8. Merged with rc. rc 2104ad7, which holds #88, #89, #97 and #98, is merged in as f98bbec, with two conflicts each resolved keeping both sides. In the AGENTS.md Pitfalls list this draft's text for a data file that is not there stays, followed by Load both on-premise extensions in one PHP, and run the mixed examples on the free data files #88's item on a wrapper from another SWIG, which replaces the item on SWIG older than 4.1. In device-detection/onpremise/readme.md Every example ends with Find out more, and gives the tagged links it needs #98's text runs the commands from device-detection/onpremise, so the web start command is php examples/onpremise/serve.php gettingStartedWeb.php, introduced as the launcher. The merge brought no change to cxx or to hash-php.i, so the committed wrapper still matches. Since Every example ends with Find out more, and gives the tagged links it needs #98 imports ExampleUtils in failureToMatch.php and manualDataUpdate.php, both now call ExampleUtils::startWithEngine.

Before you upgrade

PHP now starts when FiftyOneDegreesHashEngine.data_file names no data file, a missing file, or a file or settings the engine cannot be built from. Before, PHP stopped with exit code 134 before running any script. Now FiftyOneDegreesHashEngine::engine_get() gives null, PHP gives a warning when it starts unless the setting is left out with no file at its default path, and new DeviceDetectionOnPremise() throws an exception naming the setting. Code that calls engine_get() directly should check for null. A health check that relied on PHP failing to start should check that DeviceDetectionOnPremise can be constructed instead. The setting's name and its default path are unchanged.

Tests

device-detection/onpremise/tests/StartWithoutDataFileTests.php, added to the Unit suite in phpunit.xml, has 16 tests. ip-intelligence/tests/ServeWebExamplesTests.php, added to the Integration suite, has 2. Each test starts its own PHP with the system's own settings and a php.ini that only loads the extensions it needs, leaving out the package's php.ini and the environment variables that name a data file. The extension is loaded from a settings folder rather than the command line, so a PHP an example or a launcher starts loads it too, macOS included.

• PHP starts with exit code 0 and engine_get() null with no data file setting, an empty one, one naming a missing file, one naming a file that is not a data file, and the Lite file with settings the engine cannot use, with the warning in the last three.
• DeviceDetectionOnPremise refuses with the message for each of no setting, a missing file and a file it cannot load, and without the extension.
• Each of the five Device Detection console examples runs with nothing set, says it runs again with the Lite file, and prints its own output.
• gettingStartedWeb.php under a built-in server started with no data file answers HTTP 500 with the page naming the setting, the Lite file and the launcher.
• serve.php gettingStartedWeb.php with nothing set answers HTTP 200 with the detection ("Platform Name:"), and the IP Intelligence serve.php web with nothing set answers HTTP 200 with the lookup ("Registered Name:") and no failure message.
• The IP Intelligence serve.php web-mixed, with both extensions loaded and no data file named for either, answers HTTP 200 with both results ("Platform Name:" and "Registered Name:") and no error.

Runs

Each throwaway run builds both extensions as CI does (ci/setup-environment.ps1 then ci/build-project.ps1) on Ubuntu PHP 8.0, 8.3 and 8.5 and macOS PHP 8.5, and runs the new tests, and after the rc merge also the example tests that #88 and #98 added. The PHP that runs PHPUnit gets its own copy of the Device Detection Lite file, cxx/data/device-detection is left without one, and no IP Intelligence data file is fetched, so the examples and launchers have to download them.

• Red, run 43535109851 on tmp/fixer-dd-red at 5ef56f0, which is rc 6d95eee with only the first 15 tests and a temporary workflow. Every job printed "Tests: 15, Assertions: 32, Failures: 15." In each, 13 tests failed on exit code 134 with the uncaught StatusCodeException, the refusal without the extension on Class "FiftyOneDegreesHashEngine" not found, and the web example on "The server did not answer."
• Wrapper, run 43535118510 on tmp/fixer-dd-gen at ab33127. Its job "Generate the Device Detection wrapper" built SWIG 4.5.1 as the wrapper check does, ran generate-wrapper.sh and then --check, and uploaded the file committed as b41b100. Its preview job found 2 failures, both from ini_get_all needing the extension's lower case name, fixed in 9488de2 before the green run.
• Green, run 43536705471 on tmp/fixer-dd-green at 1191ea7, which is b41b100 with only the temporary workflow. Every job printed "OK (15 tests, 63 assertions)", and cxx/data/device-detection held the downloaded Lite file afterwards.
• Red for the launchers, run 43549652159 on tmp/fixer-dd-red-b at 2a170d2, which is b41b100 with only the launcher tests (7fa876b) and a temporary workflow. Every job printed "Tests: 16, Assertions: 67, Failures: 2." for Device Detection, the 500 page lacking the launcher command and the launcher test failing on "Could not open input file: examples/onpremise/serve.php", and "Tests: 1, Assertions: 3, Failures: 1." for IP Intelligence, on the same message.
• Green for the launchers, run 43549978800 on tmp/fixer-dd-green-b at c047d0e, which is 4207b7e, the head at the time, with only the temporary workflow. Every job printed "OK (16 tests, 69 assertions)" for Device Detection and "OK (1 test, 6 assertions)" for IP Intelligence, whose launcher downloaded the Lite file.
• Red for the mixed page, run 43564447728 on tmp/fixer-dd-red-c3 at 863f408, which is rc 2104ad7 with only ServeWebExamplesTests.php and the same workflow. On Ubuntu PHP 8.0, 8.3 and 8.5 the IP Intelligence step printed "Tests: 12, Assertions: 127, Failures: 2.", the two launcher tests for web and web-mixed failing on "Could not open input file: examples/onpremise/serve.php", while Load both on-premise extensions in one PHP, and run the mixed examples on the free data files #88's and Every example ends with Find out more, and gives the tagged links it needs #98's tests passed. Its macOS PHP 8.5 job stopped at the same rc build fault as below.
• Green after the merge, run 43564455802 on tmp/fixer-dd-green-c3 at 7514bae, which is the head c395e3f with only a temporary workflow. That workflow loads each extension from a php.ini in the settings folder, as CI does, and also runs Every example ends with Find out more, and gives the tagged links it needs #98's ExamplesWithoutSettingsTests and WebExamplesWithoutSettingsTests and Load both on-premise extensions in one PHP, and run the mixed examples on the free data files #88's MixedExamplesTests. On Ubuntu PHP 8.0, 8.3 and 8.5 the Device Detection step printed "OK (23 tests, 181 assertions)" and the IP Intelligence step "OK (12 tests, 134 assertions)", or "OK (12 tests, 138 assertions)" on PHP 8.5. The macOS PHP 8.5 job stopped while building the IP Intelligence extension from the wrapper on rc 2104ad7, with "assigning to 'zend_class_entry *' (aka '_zend_class_entry *') from incompatible type 'void *'" at ipi_wrap.cxx line 1631. That is a fault on rc that #100 fixes and this draft does not touch. The launcher tests passed on macOS PHP 8.5 in the green run for the launchers above, before Load both on-premise extensions in one PHP, and run the mixed examples on the free data files #88 was on rc. Two earlier runs for this step, 43561923735 and 43562112342, loaded the extension into PHPUnit with -d, which the PHP that Every example ends with Find out more, and gives the tagged links it needs #98's tests start does not inherit, so they are not cited.
• Full build, run 43564508164 on tmp/pr-matrix-91-c, passed. All 9 "Setting 'PullRequestSha'" lines name c395e3f, and Configure printed "Left out 5 macOS entries". On Ubuntu PHP 8.0, 8.1, 8.2, 8.3, 8.4 and 8.5, with each package's php.ini naming its data file, the on-premise Device Detection Unit suite printed "OK (88 tests, 381 assertions)" and the IP Intelligence Integration suite, which holds the launcher tests, printed "OK (74 tests, 982 assertions)", or 986 on PHP 8.3, as in job 132361027476 (Ubuntu PHP 8.3). Both counts include the example tests Load both on-premise extensions in one PHP, and run the mixed examples on the free data files #88 and Every example ends with Find out more, and gives the tagged links it needs #98 added. The Windows PHP 8.3 job skips both on-premise packages because their extensions are not built on Windows, and the Ubuntu PHP 7.4 job skips them because they require PHP 8.0. This build replaces run 43539053011 on b41b100 and run 43551019860 on 4207b7e, both of which passed before the rc merge.
• The draft's quick checks passed on c395e3f, including "Device Detection wrapper matches hash-php.i", "IP Intelligence wrapper matches ipi-php.i" and the environment variable name check.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

What remains

1. #105. The mixed console example, ip-intelligence/examples/onpremise/mixed/gettingstarted-console.php, still needs the Device Detection data file named. Its startWithEngine runs it again with the IP Intelligence data file only, so with nothing set for Device Detection, DeviceDetectionOnPremise now refuses with the message above where PHP used to stop when it started. This is read from the code and was not run. Running it with nothing set needs the IP Intelligence startWithEngine to pass the Device Detection data file too, as the launcher does for web-mixed, which is a change to the IP Intelligence example helper and is not in this draft.
2. #106. The launcher tests, like the other tests that run examples with nothing set, download the free IP Intelligence ASN file from the distributor in every CI job, because CI's asset cache does not hold it, so they can fail when the distributor refuses a free file. They are right to start from nothing, so the issue caches the file rather than changing the tests.