[jwrosewell]

PHP had no element for the derived properties that derived-properties defines, so a PHP pipeline could not offer HumanConfidence or IsGdpr, whilst .NET and Rust can. This adds a pipeline/derivedproperty package whose DerivedPropertyElement runs every script in the scripts folder of derived-properties, and tests that run every case file in its tests folder, the scripts under tests/invalid included.

Closes #74.

What the specification says

All quotes are from derived-properties at cc306386, the head of its main and the commit the new submodule pins.

docs/format-1.md, lines 4 to 6:

Every language implementation of the element (.NET, Node, Java, Python and PHP) is tested against the behaviour described here, so this document is the contract rather than a summary of one.

The one rule, lines 35 to 39:

Where every one of those properties is available, the checks and the rules run and a value is chosen. Where any one of them is not available, the script produces no value, and the message names every property that was missing along with what its source element said about each one.

When a property is absent, lines 451 to 455:

Nothing is evaluated. The output property is present in the element data with no value, and reading it raises the language's existing no value error, which is AspectPropertyValue with a NoValueMessage in .NET and the matching mechanism in each other language. The message names every absent property rather than only the first one found.

Lines 491 to 497, on an element added after a source it reads:

Where an element runs is not judged at build, since a pipeline holding elements that run in parallel reports them in an order that does not say what ran before what. An element added after a source it reads therefore builds, and produces no value on every request, with this message naming the property. An implementation should log that once rather than on every request, because it will otherwise repeat for the life of the process.

Writing YAML, lines 562 to 570:

YAML 1.1 parsers, which several languages still ship, read yes, no, on and off as booleans ... The reference tools in this repository use the YAML core schema, where yes stays the string "yes"

Two places where the languages are allowed to differ, lines 590 to 605, being the line a fault names, and how strictly a JSON script is read:

Text that is broken is reported as broken either way and only the wording of the fault differs, so no script is valid in one language and invalid in another.

docs/testing.md, lines 3 to 6:

every 51Degrees language implementation runs the same case files, so a case written here becomes a test in each language as that language is built, starting with .NET and followed by Node, Java, Python and PHP.

Line 177 and 196 to 197:

Every property that could not be read has to be listed, so a case naming one of two unreadable properties fails

tests/invalid/ holds scripts that every implementation must refuse, along with what each refusal has to say.

docs/authoring.md, lines 365 to 384 asks for one information line per script at build naming "the script, its version, the format number, where the script came from (a built in name, a file path or code) and the output property", and at debug level "the compiled model as canonical JSON". Lines 441 to 442 add a warning for a deprecated script. Lines 452 to 464 describe the check when the element is added, where "a source property that no earlier element supplies" and "two elements in one pipeline producing the same derived property name" fail the pipeline build. The word "earlier" there is at odds with format-1.md lines 491 to 494 above, and .NET and Rust accept an element that supplies the property wherever it sits, so PHP follows the format reference, which calls itself the contract.

README.md, lines 80 to 109 names the build parameters Scripts and ScriptFiles, each "one string with commas between the names", where "a name that holds a comma is wrapped in double quotes".

What changed

The reference implementations read were .NET, pipeline/DerivedProperty of 51degrees-dotnet, and Rust, the derived-properties crate of 51degrees-rust. The PHP code ports the reference tools the specification follows, tools/parse.mjs, validate.mjs, evaluate.mjs and canonical.mjs, with the same fault paths and messages.

1. The submodule. .gitmodules adds derived-properties at the repository root with the relative URL ../derived-properties, pinned at cc306386, beside javascript-templates.
2. The package carries the scripts. Packagist serves an archive of a tag and an archive has no submodules, which is why pipeline/core commits a copy of the JavaScript templates. pipeline/derivedproperty/scripts is the same kind of copy of the submodule's scripts folder. ci/update-packages.ps1 of the package moves the submodule to the head of derived-properties and refreshes the copy, as pipeline/core/ci/update-packages.ps1 does for the templates, and BuiltInScriptsTests::testTheCopyIsTheScriptsFolderOfTheSubmodule fails when the two differ. The package is not on Packagist, as it has no mirror, like pipeline/translation.
3. Reading YAML. PHP has no YAML reader, and a generated form of the built in scripts would not be enough, because the format accepts YAML for script files and for script text held by code, and the rejection cases include faults found while reading YAML, such as text that is not YAML and keys that differ only in case. So the package requires symfony/yaml ^5.4 || ^6.4 || ^7.0 || ^8.0, the Symfony project's YAML reader, which is plain PHP and needs no extension. It reads yes and no as strings, as the core schema the reference tools use does, and its objects for mappings keep an empty mapping apart from an empty list. Composer picks the newest that the PHP of each job supports. JSON scripts are read with json_decode, the strict reading the specification allows.
4. The element. DerivedPropertyElement writes one AspectPropertyValue per script under derived, through DerivedPropertyData, where names are matched without regard to case. A script with a source property missing has no value, and reading value raises the message naming every missing property, which is the PHP pipeline's no value error. It reads a source as PHP elements hold it, by its lower case name first, carrying the source's own no value message or the message of the exception an engine raises for a property it does not have, and taking the heaviest of a weighted list in the value and rawweighting shape IP Intelligence gives. A script reads what an earlier script of the element wrote. Missing element data is logged once by each element. PHP keeps nothing between requests, so where the pipeline is built on every request, as it usually is, each request logs it again.
5. Configuration. The constructor takes the Scripts and ScriptFiles settings, so PipelineBuilder::buildFromConfig builds the element, and DerivedPropertyElementBuilder offers addBuiltInScript, addScriptFile and addScript in code, as .NET's builder does. Every fault in every script is reported in one ScriptValidationException. A setting the element does not have is refused, so a setting written wrongly cannot leave its scripts out without a word.
6. When the pipeline is built. The element logs the line and the canonical JSON authoring.md asks for, and refuses the pipeline with the same messages as .NET and Rust where nothing supplies a source property or another element writes the same derived property. PHP builds the pipeline on every request, and pipeline/core/tests/PropertyDatabaseTests.php holds that building one asks no element for its properties. So only the elements whose data key a script names are asked, an element with findProperty, as both on-premise engines have, is asked for that one property, and an element listing no properties may supply any, because a cloud engine learns its properties from its first response (CloudEngine::processInternal).
7. Metadata. Each property's metadata has the keys the other PHP elements give, being name, type (named as on-premise Device Detection names types), category, description, available and isList, followed by the rest of the Output block.
8. Wiring. The root composer.json installs the package from its directory, and ci/packages.ps1 lists it as a Root package with no mirror, so the full build runs its Unit suite in every job. On the PHP 7.4 job it is left out, as it needs PHP 8.0. The README and AGENTS.md describe the package and the copy.

The tests, in pipeline/derivedproperty/tests:

• SharedCasesTests runs the 69 value cases (57 for HumanConfidence and 12 for IsGdpr), each through a real pipeline with a source element giving the case's properties and the script the package carries, and the 29 rejection cases, each refused with a fault at every expected path and mentioning every expected fragment. A missing value must name exactly the properties the case lists.
• BuiltInScriptsTests checks the copy against the submodule and builds one element with every built in script.
• PipelineCheckTests follows the Rust crate's tests/pipeline_check.rs, with one more test that an element with findProperty is never asked for every property.
• ElementTests and ScriptTests cover reading the values PHP elements give, logging, settings, metadata, the JSON bundler, YAML and JSON, type inference and conversion.

Runs

Each throwaway run uses a workflow that installs the package with Composer against pipeline/core beside it and runs its Unit suite on Ubuntu with PHP 8.0, 8.1, 8.3 and 8.5 and on Windows with PHP 8.3. A sixth job archives pipeline/derivedproperty with git archive, as a mirror's tag would be served, installs it from Packagist, and runs the built in scripts.

• rc at 6d95eee, which holds #87 and #77, is merged in with a merge commit, 787e154, with no conflicts. Its tree is the one git merge-tree gives for 9a22bd6 and 6d95eee, so nothing was edited in the merge.
• Red, run 43500164432 on 91a16e27, failure. That commit is the first head's green commit 31b45a18 with pipeline/derivedproperty/src removed. Every test job failed, PHP 8.5 with "Tests: 52, Assertions: 5, Errors: 49, Failures: 3", each error a class of the element not being found and each failure a test that expected an exception from the element, and the package job failed with Class "fiftyone\pipeline\derivedproperty\BuiltInScripts" not found.
• Green, run 43526611436 on 3e7d08ca, success. That commit is this pull request's head 86543e8 with the throwaway workflow added, and nothing else. Every test job ran "OK (52 tests, 140 assertions)", with symfony/yaml 5.4.56 and PHPUnit 9.6.39 on PHP 8.0, 6.4.50 and 10.5.67 on 8.1, 7.4.23 and 12.5.44 on 8.3 (Ubuntu and Windows), and 8.1.11 and 13.4.7 on 8.5. The package job listed scripts/HumanConfidence.yaml and scripts/IsGdpr.yaml in the archive, installed 51degrees/fiftyone.pipeline.core 4.5.207 from Packagist, built HumanConfidence 0.8.0 and IsGdpr 0.1.0 as built in HumanConfidence and built in IsGdpr, and answered IsGdpr true for FR.
• The last commit, 86543e8a, changes only the comment and the message of the missing element data error, and the test of it, so the element behaves as before and there is no red run for it. testElementDataNotThereIsLoggedOnceByEachElement now checks that one element logs the message once over two requests, and that a second element logs it again, as a PHP web server that builds the pipeline per request does. It passes in the green run above.
• Full build of the head 86543e8a, run 43526636091 on tmp/pr-matrix-89-b, started with tools/pr-matrix.py without macOS, success. All 8 build jobs passed, Ubuntu PHP 7.4, 8.0, 8.1, 8.2, 8.3, 8.4 and 8.5 and Windows PHP 8.3, and the two performance comparison jobs were skipped. Every one of the 9 Setting 'PullRequestSha' lines gives 86543e8, and Configure printed "Left out 5 macOS entries". Each job from PHP 8.0 installed the package from its directory and ran its Unit suite with "OK (52 tests, 140 assertions)". The PHP 7.4 job left the package out of the root composer install and skipped its tests, because it requires PHP 8.0.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

What remains

• Building an element with both built in scripts reads their YAML, which the green run of the first head, run 43500152072, measured at 1.0 to 1.25 ms on Ubuntu and 2.5 ms on Windows. PHP builds the pipeline on every request, so a pipeline holding the element pays that on every request. #96 carries the built in scripts in a compiled form.
• On-premise Device Detection reads a Boolean property with getValueAsBool, which answers value == "True" (ResultsBase.cpp lines 118 to 140), so a Boolean property stored as Unknown reaches PHP as false. HumanConfidence compares device.IsVisible and device.HasWebDriver with "Unknown", and where the data file types those properties as Boolean that comparison is never true through that engine. That needs a data file to check, so it was not checked here. .NET reads the stored string where a script compares a property with text, and #95 does the same for PHP once the data files are checked.
• The consumer tests of derived-properties build only .NET. derived-properties#18 adds a job for PHP.
• format-1.md asks for missing element data to be logged "once rather than on every request". The element keeps that record for as long as it lives, which in a PHP web server is one request, so there it is logged on every request. PHP static state does not outlive a request, so there is nowhere in the package to keep it longer.
• .NET also lets a script replace a property of another element, by an Output.Name naming that element. Format 1 as published at cc306386 asks Output.Name to match ^[A-Za-z][A-Za-z0-9]*$, so PHP follows the format and has no such scripts.
• #37 replaces the package table in ci/packages.ps1 with ci/packages.json. Whichever of the two merges second adds pipeline/derivedproperty to the other's form of the table, with no mirror and not published.
• The throwaway branches tmp/php-74-quick, tmp/php-74-red, tmp/pr-matrix-89, tmp/php-74-green-b and tmp/pr-matrix-89-b have been deleted. The runs above stay readable without them.